Now, any technical team that is worth its salt knows what the problems in their infrastructure are, and will have plans to take care of them. Where things break down at an ISP are either that the tech team isn't good, or the management doesn't listen to them, or there isn't money. It wasn't true which of the latter two was the case at AONL when Scott arrived, but the tech team had changes they wanted to make, and the first thing Scott did was just to facilitate having those happen.
One of the biggest problems to be resolved in the first two months was with Asia Online's backbone network. The backbone of an ISP is composed of the connections to bigger ("upstream") ISPs, and the size of the ISP's backbone, and the degree of fullness, is what will define the experience the users of that ISP have when they access external resources on the Internet.
Part of what should be happening in a well-run network is that the staff has some way to monitor what is going on in the network. Usually, because ISPs are relatively poor, this is done with the free tool MRTG, written in part by Scott's acquaintance Dave Rand. MRTG, for Multi-Router Traffic Grapher, is a system that probes the switches and routers that direct traffic within an ISP's backbone; it then stores the data and provides a set of web pages with graphs of the data. In short, MRTG enables you to look at the total volume of traffic going over any given part of your network.
Asia Online was in fact using MRTG, and they had graphs of what was going on. What took a long while to sort out, possibly because of language differences, and possibly in part due to differing cultural assumptions, was that even though it didn't look that way on the MRTG graphs, the traffic was hitting an arbitrary limit at 50% of the stated capacity of the backbone. It turned out that the backbone connection in question was only guaranteed up to the 50% level, and anything above that was catch-as-catch-can - not guaranteed at all. Once the facts were established that support that, it became obvious that the traffic level had a "haircut" at the 50% level, such that the graph ramped up in the morning, then went flat at 50% all day long, and finally dropped back down around midnight.
One of the additional minor bits of information that Scott discerned from looking at these MRTG graphs was that Hong Kong was a late night city compared to San Francisco. The evening peak at Asia Online started later and ended later than it had at Whole Earth Networks, with significant traffic going on until around 2AM. This was something that was evident as well in walking around Hong Kong on a Sunday morning - if you went out early to avoid the heat, nothing was open. The restaurants might be open for breakfast, but the various small shops largely didn't open until between 11AM and noon.
The reason that all of this technical fiddling around was important to Asia Online - important enough to import Scott all of the way from California to deal with it - was expressed succinctly in another of Kevin Randolph's adages: "You can't fill a bucket if the water leaks out the bottom as fast as you pour it in the top." Having a network that was filling up in the morning meant that from that point until the evening, everybody had trouble getting out on the Internet. They had slow performance, they became unhappy, and eventually they stopped being customers of Asia Online. It didn't matter how hard the company might work on marketing itself - Kevin's forte - or how quickly they acquired new customers, if they lost customers just as fast because the service stank.
There were other technical tweaks that Scott oversaw and caused to happen - he
was assigned a modest $400k budget to ensure that the service improved, and he
spent the money and the service did in fact improve. There were other changes
that Kevin made to customer acquisition, but underneath it all, improving
customer retention helped move Asia Online from shrinking to growing.
Posted by scott at February 9, 2006 03:46 PM