February 01, 2006

Page 20

Hong Kong International Airport at Chek Lap Kok opened on July 6th, 1998. Scott was passenger number thirty thousand-and-something and has the certificate to prove it. The flight from San Francisco to Hong Kong took some twelve to fourteen hours, and sleeping in coach class was next to impossible. Furthermore, this was Scott's first international overseas trip - he had been overland to Canada a couple of times and to Tijuana twice, but it's hardly the same thing. He had gotten a passport while working at Vixie Enterprises but hadn't used it, so he arrived in Hong Kong clutching his not-quite-new but otherwise virginal documents.

Now, HKIA at CLK (or just CLK, for short) is a big airport. It may not be the biggest in the world, but it's big enough to be extremely confusing to a sleep-deprived newbie making his first international flight. Kevin had briefed Scott that there was a train connecting the airport to the actual city, and they planned that Kevin would meet Scott at the airport, but they both knew that the exigencies of business might disrupt that plan.

So, on arriving at CLK, the normal process is that you stumble off the plane in a stupor (the stupor is a requirement on all flights over more than six time zones or more than 12 hours in duration) and follow the herd through the lower-level hallway. CLK is laid out on two levels, you see - the upper level, with the vaulted ceilings, huge windows and award-winning architecture is for departures. The lower level, with standard-height ceilings and much smaller windows is for arrivals. You trundle along, with your carry-ons until you reach an escalator, and following the crowd, you descend to a platform. The next thing you know, you are on a train, standing-room-only, of course, and you're wondering what happened to immigration and customers, baggage claim, and all of the other niceties and why you are already on the train to the city.

The stupor is very important, you see. It keeps you from realizing, on that crucial first trip, that you can't possibly already be on the train to the city, but it also keeps the panic from escalating very far before the train arrives at the main platform and you step back off. From here, if you've done the least bit of traveling, things begin to make a little more sense. You collect your bag from the enormous carousel and take it over to be inspected, though you more usually get to be waved through. It's then onward to the immigration line. After you pass customs, you loop around a dividing wall and come up to the rope line separating the secure part of the airport from the public section. That's where Kevin was waiting for Scott, and it was one of the happiest sights Scott had seen in quite some time.

Geographically, Hong Kong is divided, like Gaul, into three parts. Originally, the British pried Hong Kong Island from the Chinese. It was on a "lease" for 99 years in perpetuity, meaning that the lease automatically renewed for as long as Britain had bigger guns than China.

Some while afterwards, the British realized that HK Island wasn't very viable. They then extracted two more bits of land from the Chinese - first, the land opposite the island on the mainland, which is Kowloon; and secondly, a grab bag of islands and bits of the mainland that came to be known collectively as the New Territories, or NT. The NT was Britain's final attempt to make HK self-sufficient in terms of things like food and water. By the 1980s, there was a problem. It wasn't so much that the Chinese had much bigger guns than they had 100+ years before, though they did. It wasn't that the British had a case of collective remorse over colonialism, though they may have. It was more a practical matter. It seems that the NT wasn't on an automatically renewing lease, and the Chinese were being stubborn about renewal. They were willing to let the lease on HK Island and Kowloon renew - at least, they said they were - but they weren't about to let Britain renew the lease on the New Territories. Furthermore, Hong Kong as a total entity was already dependent on the Chinese mainland for water and power; without the New Territories, the remaining areas, which contained something like 20% of the land and 85% of the population, would be totally unsupportable.

Ultimately, of course, what happened is that China and Britain negotiated the return of Hong Kong to Chinese control, with HK becoming a "Special Administrative Region" or SAR - "one country, two systems." In practical terms, for an American expatriate in the late 20th century, it meant relatively little. The uniformed Chinese people with machine guns worked for the People's Republic of China instead of Great Britain, but they still had machine guns, and at that point, who paid them made relatively little difference.

The geography of Hong Kong meant somewhat more than the politics. Hong Kong is both a territory - the SAR - and a city, with the city located on the north shore of Hong Kong Island and the south shore of Kowloon peninsula. If you meet someone from Hong Kong, the usual question is, "Hong Kong side or Kowloon side?"

Chek Lap Kok was a small island off of Lantau Island, in the western part of the New Territories. CLK Island was extended with landfill and the airport was built on top of it. A new freeway and rail line was built along Lantau Island to connect the airport to Hong Kong Island. The rail line, the Airport Express, also had a semi-local spur route that branched off to a "New Town" that was built near the airport to house the airport's workers.

What all of this meant is that CLK was very well-connected with Hong Kong proper, despite being out in the sticks. Your travel options included the Airport Express, which was a very nice train, as well as several options via road, such as taxi, private car, or any number of airport buses.

Posted by scott at 02:55 PM

February 03, 2006

Page 21

Now, the urban portion of Hong Kong Island is broken into a number of districts. The downtown-est downtown portion is Central; going eastward from Central you hit the Admiralty district, then Wan Chai, Causeway Bay (Tung Lo Wan) and North Point. These districts for the most part defined Scott's sphere of activity in Hong Kong. The Airport Express connects to Central, where there is a pedestrian tunnel running between the Airport Express station and the Central stop on the MTR (Mass Transit Railway), Hong Kong's subway system. The tunnel is fairly long, and entails not one but two (or three?) moving slideways to cover the distance. Even with the mechanical assistance, dragging one's luggage on and off those trains and through all of those stations, at what one's body is insisting is 4AM or thereabouts, is no picnic. Still, following Kevin's lead, that is exactly what Scott did. They switched trains at Central and took the MTR east to Fortress Hill station. It might be that there is in fact a hill at Fortress Hill MTR - you never know - but whatever it is, the Fortress Hill MTR station is rather deep underground, and the escalator from the platform to the mezzanine level is quite fast. Once you reach the mezzanine level, you get to haul your bags up one more flight of stairs - no escalator - and finally, you will emerge at Street Level.

Scott's first real memory of Hong Kong proper, then, is of the street just outside of Fortress Hill MTR, near the border between Causeway Bay and North Point. It was hot, humid, dark out but brightly lit and noisy, and the traffic lights made the most unearthly rattling sound. In all, it was a little bit of hell.

Where Scott and Kevin stayed during those first months at Asia Online was at the Newton Hotel. The Newton was quite reasonable by Hong Kong standards and very convenient to the Asia Online office, just 5 to 10 minutes on foot, depending on just how hot and sweaty you wanted to get.

The price for staying at the Newton was that the rooms, with a little glue, would have served quite nicely as postage stamps. They all came in the same size (ultra-small), with a choice of one bed or two, and they faced either right or left. As a Californian, Scott was quite grateful that they had a non-smoking floor, as tobacco was quite popular in HK at the time.

Anyhow, the room was slightly larger than the double bed on two sides - just enough to sidle around it. The third side featured a built-in desk with a cube refrigerator (full of the usual honor-bar crud) and a 13-inch television. The TV had one or two english-language channels, a bunch of Chinese channels, and some other miscellaneous bits. You could also "preview" the soft-core down about the same way as the regular channels.

There wasn't anything resembling a dresser or chest of drawers. In the closet, which was about the size of a small guest closet in an older American home, there was one drawer below the hanging area, and one shelf above the hangar rod. You also had a little floor space for your shoes. Lastly, the window was a bit of a bay window and there was a good-sized shelf there, which made a decent space for storing suitcases.

The bathroom was likewise compact, though not outrageously so. Hong Kong people are fond of "on demand" water heaters, so after a hot shower you would find that the mirror was strangely free of steam and quite warm to the touch. Presumably, the on-demand heater was behind the mirror.

The view from the Newton is singularly uninspiring - a small slice of Victoria Bay and Kowloon. One of the things about Hong Kong is that all of the exciting architecture that you see in media is in fact there, and it's all on the Hong Kong side. If you're on the Hong Kong side, what you see is Kowloon, which is heavily residential and hence largely socialist worker housing. If you want to enjoy looking at something like I. M. Pei's famous Bank of China building, you really want to live in Kowloon, looking back at HK Island.

For those who aren't familiar with it, Pei's BoC building looks like a giant knife, slicing back into the hills overlooking the bay. Specifically, the knife is slicing at the British Governor General's mansion...a bit of feng shui that the locals rather delight in, quite understandably.

Posted by scott at 04:04 PM

February 06, 2006

Page 22

Asia Online was located in Citicorp Centre, occupying the 23rd floor. This was a fairly posh bit of real estate, though as it is in Causeway Bay rather than in Central, not outrageously so to the locals. Citicorp Centre, located on Whitfield Road, is about a block from Victoria Park and hence fairly visible because it's the first tall building on the east side of the park. It is standard smoked glass, with bezeled corners and an angular constriction about midway up. The bezeled corners meant that the corner offices were basically triangular and a bit cramped.

Scott and Kevin occupied side-by-side offices on the side of the building overlooking the park and marina. The marina was quite a peculiar sight, as it was roughly rectangular with a diagonal aisle. On one side of the aisle were the western-style boats, which were uniformly white. On the other side of the aisle were the Chinese junks, which were the brown of the woods used in their construction. From above, the segregation by color was quite striking.

Hong Kong SAR is about 85% parkland, so to say that Victoria Park was HK's largest park would be a misnomer. Still it is a good-sized urban park and makes a significant break in the cityscape. That open space also gives the occupants of Citicorp Centre a good view.

The assignment Kevin had taken and that Scott was now part of was to look AONL over, determine if it could be fixed up, what was needed, and if the owners approved, actually do the fixing. Despite the name, what Asia Online was in fact was just a mid-sized local Hong Kong Internet Service Provider. They sold the usual menu of services (dial-up, leased line and hosting) to the usual sorts of customers, individuals and small businesses for the most part. They had somewhere around 18,000 dialup customers, perhaps a hundred or so leased-line customers (much higher speed and much higher revenue, the cream of the customer crop) and a relatively small hosting facility up there in their expensive real estate. As a small capper, in order to reduce the load on the air chillers in their data center and to control the electric bills, they had put tinfoil up over the windows in that part of the building. This wasn't exactly unreasonable, since it did get pretty warm in the sunny spots, but it was a bit tacky.

The first order of business for Scott was simply to get oriented. Hong Kong is eight or nine time zones away from the US, depending on the time of year, and it takes about a week just to adjust to the time change. In addition, there was a whole new city-state to explore. As an example, after Scott arrived in Hong Kong, he was so disoriented in his sense of place that he had his directions reversed and though he needed to turn left at the main road to head toward Central, when in fact he needed to turn right. Exploring his surrounds on the weekends, it took him two to three weeks to sort just this small item out. Eventually, combining riding the rails with walking, he walked the entire distance from Central to North Point. Given the HK summer weather, though, he only walked the distance from one station to the next on each weekend day that he went out on this trek.

Orienting himself with Asia Online was no less time-consuming. Most of the HK Chinese staff that Scott worked with had perfectly serviceable English. However, they have a sort of cultural shyness, especially the technical staff, and especially about their language, so they were painfully uncomfortable with the sort of casual converstation that California's high-technology business is built on. Ultimately, Scott's habit of using e-mail for even trivial matters came to his rescue, as the staff let him know that with e-mail, they could read and re-read his requests until they understood them.

Initially, Scott's ambit was simply as a consultant to Kevin, but after a few weeks it became time to start to change things up. As the first step, three formerly separate technical groups - the team that ran the systems and networks, the technical support organization, and the internal MIS group - were merged together into a Technology department. Scott was then announced to the merged group as their interim vice president. This gave Scott a team of some 15 or so people, including three subordinate managers.

Posted by scott at 03:54 PM

February 09, 2006

Page 23

Since this was a consulting gig with a time-limited contract - 8 weeks with possibly a second 8 weeks - it wasn't going to be Scott's job to run this team indefinitely. He had two tasks; first, to identify major technical or personnel problems and fix them; second, to hire a full-time manager for the organization.

Now, any technical team that is worth its salt knows what the problems in their infrastructure are, and will have plans to take care of them. Where things break down at an ISP are either that the tech team isn't good, or the management doesn't listen to them, or there isn't money. It wasn't true which of the latter two was the case at AONL when Scott arrived, but the tech team had changes they wanted to make, and the first thing Scott did was just to facilitate having those happen.

One of the biggest problems to be resolved in the first two months was with Asia Online's backbone network. The backbone of an ISP is composed of the connections to bigger ("upstream") ISPs, and the size of the ISP's backbone, and the degree of fullness, is what will define the experience the users of that ISP have when they access external resources on the Internet.

Part of what should be happening in a well-run network is that the staff has some way to monitor what is going on in the network. Usually, because ISPs are relatively poor, this is done with the free tool MRTG, written in part by Scott's acquaintance Dave Rand. MRTG, for Multi-Router Traffic Grapher, is a system that probes the switches and routers that direct traffic within an ISP's backbone; it then stores the data and provides a set of web pages with graphs of the data. In short, MRTG enables you to look at the total volume of traffic going over any given part of your network.

Asia Online was in fact using MRTG, and they had graphs of what was going on. What took a long while to sort out, possibly because of language differences, and possibly in part due to differing cultural assumptions, was that even though it didn't look that way on the MRTG graphs, the traffic was hitting an arbitrary limit at 50% of the stated capacity of the backbone. It turned out that the backbone connection in question was only guaranteed up to the 50% level, and anything above that was catch-as-catch-can - not guaranteed at all. Once the facts were established that support that, it became obvious that the traffic level had a "haircut" at the 50% level, such that the graph ramped up in the morning, then went flat at 50% all day long, and finally dropped back down around midnight.

One of the additional minor bits of information that Scott discerned from looking at these MRTG graphs was that Hong Kong was a late night city compared to San Francisco. The evening peak at Asia Online started later and ended later than it had at Whole Earth Networks, with significant traffic going on until around 2AM. This was something that was evident as well in walking around Hong Kong on a Sunday morning - if you went out early to avoid the heat, nothing was open. The restaurants might be open for breakfast, but the various small shops largely didn't open until between 11AM and noon.

The reason that all of this technical fiddling around was important to Asia Online - important enough to import Scott all of the way from California to deal with it - was expressed succinctly in another of Kevin Randolph's adages: "You can't fill a bucket if the water leaks out the bottom as fast as you pour it in the top." Having a network that was filling up in the morning meant that from that point until the evening, everybody had trouble getting out on the Internet. They had slow performance, they became unhappy, and eventually they stopped being customers of Asia Online. It didn't matter how hard the company might work on marketing itself - Kevin's forte - or how quickly they acquired new customers, if they lost customers just as fast because the service stank.

There were other technical tweaks that Scott oversaw and caused to happen - he was assigned a modest $400k budget to ensure that the service improved, and he spent the money and the service did in fact improve. There were other changes that Kevin made to customer acquisition, but underneath it all, improving customer retention helped move Asia Online from shrinking to growing.

Posted by scott at 03:46 PM