30 June, 2007

Datacenter Confidential #3

"Fucking Bill Clinton," said the boss man, who we will call G.

The date was about a week after September 11, 2001. By then, everyone had returned, cautiously, back to work. On my walk from the Embarcadero to under Coit Tower, I would glance up at the skyscapers, wondering how they too could come down in a cloud of smoke, debris and pulverized human beings. Commercial aviation had just started back up, and my neck twisted with the strain of every turbo-jet above, as if flying were brand new again, and I gazed in wonder as heavy, fuel and passenger laden tin tubes blasted unbelievably through the sky.

I explained to G that I felt that, although the CIA had been dismantled somewhat during the 8 years of Bill Clinton's presidency, the process had actually been going on for much longer -- since before Carter, to be precise. I said I felt that because the Cold War was ostensibly over, it was reasonable to start dismantling the apparatus of the Cold War. And I mentioned that Clinton was fairly diligent in pursuing bin Laden, yet all we heard from Bush 43 was rhetoric aimed at Saddam Hussein, whom I felt wasn't a threat (and I was right!).

I knew then that Bill Clinton was not to blame, just as I had known by 8am the previous Tuesday that Osama bin Laden was behind the attack, 11am Eastern Daylight Time. I had an inkling that the idiot king, bestowed upon us by the Supreme Court late in 2000, may be complicit -- our president whose initial blunders were numerous and embarrassing, for me the most memorable being the China spy gaffe. Only later would my suspicions about the level of incompetence be validated, and then some.

Fast forward to another Tuesday: namely last Tuesday.

"I don't care about politics," said G, "unless they effect Chinese trade." J'accuse, G! We finished our beers and made way back to the apartment to continue discussing international trade, tech industry ex-patriotism and various subjects over dice, Jim Beam and Sierra Nevada Pale Ale.

Don't think I'm going anywhere with this, I'm just "introducing" G as a "character" in my "richly coloured world of semi- biographical 'fiction'"(*).

(*) I only expect the lawyers to believe any of this is fiction.

G was brought on as a CTO in early 2001, and everyone understood that this was the hatchet man. G was Blake, but dressed up with a Columbia MBA, and experience as an Oracle DBA. Compare his first words in his first group engineering meeting with us with the words from Mamet:

G: I read the code last night.
[Looks around the room, making eye contact with everyone]
G: After I was finished throwing up on my shoes, I honed in on a few items that we need to fix right away.
Hapless Engineer: Well, sure there have been some items that we've had to work around, legacy issues, etcetera.
G: The code is garbage.


Blake: 'The leads are weak.' Fucking leads are weak? You're weak. I've been in this business fifteen years.
Moss: What's your name?
Blake: FUCK YOU, that's my name!! You know why, Mister? 'Cause you drove a Hyundai to get here tonight, I drove a eighty thousand dollar BMW. That's my name!! (to Levene) And your name is "you're wanting." And you can't play in a man's game. You can't close them. (at a near whisper) And you go home and tell your wife your troubles. (to everyone again) Because only one thing counts in this life! Get them to sign on the line which is dotted! You hear me, you fucking faggots?
(Blake flips over a blackboard which has two sets of letters on it: ABC, and AIDA.)
Blake: A-B-C. A-always, B-be, C-closing. Always be closing! Always be closing!! A-I-D-A. Attention, interest, decision, action. Attention -- do I have your attention? Interest -- are you interested? I know you are because it's fuck or walk. You close or you hit the bricks! Decision -- have you made your decision for Christ?!! And action. A-I-D-A; get out there!! You got the prospects comin' in; you think they came in to get out of the rain? Guy doesn't walk on the lot unless he wants to buy. Sitting out there waiting to give you their money! Are you gonna take it? Are you man enough to take it? (to Moss) What's the problem pal? You. Moss.
Moss: You're such a hero, you're so rich. Why you coming down here and waste your time on a bunch of bums?
(Blake sits and takes off his gold watch)
Blake: You see this watch? You see this watch?
Moss: Yeah.
Blake: That watch cost more than your car. I made $970,000 last year. How much you make? You see, pal, that's who I am. And you're nothing. Nice guy? I don't give a shit. Good father? Fuck you -- go home and play with your kids!! (to everyone) You wanna work here? Close!! (to Aaronow) You think this is abuse? You think this is abuse, you cocksucker? You can't take this -- how can you take the abuse you get on a sit?! You don't like it -- leave. I can go out there tonight with the materials you got, make myself fifteen thousand dollars! Tonight! In two hours! Can you? Can you? Go and do likewise! A-I-D-A!! Get mad! You sons of bitches! Get mad!! You know what it takes to sell real estate?
(He pulls something out of his briefcase)
Blake: It takes brass balls to sell real estate.

Needless to say, the engineering department shrank significantly from that point forward. Some of the programmers where nice people, but a lot of them were dead weight. The cuts were needed, and as Blake says at the end of his little speech in Glengarry Glenn Ross "a loser is a loser." My coworker, ostensibly the lead systems administrator, once told me that G would see a priest after each round of layoffs. I don't know if that's true or not, but it certainly adds to the legend.

One final anecdote: early on during G's first week, a group of H1B workers from China (Hong Kong, mainly) had gathered in a hallway to discuss, worriedly, in Mandarin their fate, and to suss out just what this new G fellow was up to. At the same time, G, my co-admins and I come in from the elevator after lunch to catch the tail end of the Chinese programmers discussion. After a moment, G breaks in to the programmers' discussing in pitch perfect Mandarin, much to everyone's surprise.

29 June, 2007

Movie Review: Syriana

This movie is pretty great, so far. I'm about halfway through it, and I wish it were, believe it or not, longer. I think that the material would make for a terrific television series -- one that no one, not even HBO, especially not even HBO, post Albrecht, would produce.

If George Clooney and Matt Damon (and the supporting cast, including Chris Cooper and Jeffrey Wright) wanted to serialize it, I would watch it. Me, and three other people.

I am not sure there is a place for a nuanced look at the politics of petroleum, at the real, live human beings that inhabit places like Iran and Iraq and Pakistan, in a country which produces thinkers like this genius, the American version of the Taliban, but far, far scarier (we have infinitely more resources to wage rageful destruction on our supposed enemies).

In Syriana, a group of foreign workers in, presumably Saudi Arabia, attend a Madrasa and are treated to a free feast of flatbreads, dates, lamb kebobs and pilafs. The cleric, an Arab, instructing a group of young Pakistanis and Afghans, instructs the foreigners on religious thought -- turn your back on the modern secular world. Look, he says, at the Americans and the decline of their society, esteem, reputation and wealth. Draw the line from their godlessness to their problems. Surely, he says, this is Allahs will. But the Pakistanis and other foreign workers face unemployment, poverty and deportation as guests in the fertile crescent. Hope is what religious instruction offers these people, hope and hate. It is the same as in Kansas or Utah, California or Indonesia. And the same thing that is "wrong with Kansas" is surely wrong in all of Allah's vast kingdom: poor people, fed the rhetoric of hope and hate, used as pawns in a power struggle designed to keep these same people as mules for those in power, be they the emir or a president.

Siddig El Tahir El Fadil El Siddig Abderahman Mohammed Ahmed Abdel Karim El Mahdi, better known as Siddig El Fadil or even better known as Alexander Siddig (and known to me as Dr Julian Bashir) makes an appearance as Prince Nasir Al-Subaai. He also makes appearances in "24", Fox's near-pornographic Islamofascist hate-fest (a show where the fake Fox News broadcasts are hard to distinguish from the networks' "real" news broadcast), he appears as Khalid Sheik Mohammed on a TV movie called "The Hamburg Cell" and a number of other credits that appear to say the same thing: if you name is like Siddig El Tahir El Fadil El Siddig Abderahman Mohammed Ahmed Abdel Karim El Mahdi, you will get roles on TV as a terrorist. Prince Nasir is not that, nor was Julian Bashir, a positive role for Siddig on a science fiction franchise famous for its African-american communications officer, Vulcan science officer, Klingon security chief, a gay Japanese man, black, female and (gasp) British commanders and captains living in the rainbow colors of Benneton Federation future.

Well, the future looks sexier in Farscape, Firefly and Battlestar Galactica (if its not actually the past), but Star Trek could be worse -- it could be Star Wars, where politics and dialog hover down at the George W Bush level of 8-year old "hate our freedom" discourse.

Which comes back to my original point, and plaintive plea: George Clooney, if you are reading this (and I know you aren't), stop what you are doing and pitch Syriana as a series. A multi-year, multi-arc, 50+ hour television drama. Grab David Chase or, better, David Simon, and Robert Baer and Stephen Gaghan and find out who the new Chris Albrecht is and pitch it, baby, pitch it.

The Wire ends next year. What will fill the void? You can take 5 years off and film a series, can't you? America needs to know about this, think about this. Clooney, we need you. And, as a way to send a nice "fuck you" to Fox News, you can use Keith Olbermann for incidental background news clips. Think about it, it would be bril.

Glenn Greenwald is a genius

Only Glenn Greenwald could elegantly combine a political critique of dick cheney and family guy in this post.

Bra-vo, Greenwald, bravo..

22 June, 2007

datacenter confidential #2

I have become used to a certain degree of derision from Steve, the datacenter facilities manager. Here is a man in his mid-50s who has probably seen and done anything and everything I am likely to accomplish in my career 10 times over.

But there was something different in his eyes when both he and I realized that my predecessor had foolishly plugged in a 15-amp remote power controller (RPC) into a vertical power strip plugged into a 20-amp power circuit.

"That thing could catch fire," Steve told me, in the shrillest of tones.

I looked back at him confused and a little hurt. "What amperage do we require you to use on a 20-amp circuit?" continued Steve.

"15 amps, 80%" I repeated, from rote.

"What's 80% of 15 amps?"

"12 amps?" I replied using quick math.

The circuit that 15-amp RPC was plugged into was drawing nearly 11 amps. It dawned on me that if I plugged anything else into that RPC there was a good chance that, at best, I would blow the breaker inside the RPC and, at worst, it could catch fire.

This RPC, which I had been haphazardly plugging things into for over a year and a half, was some sort of half-assed jury rig -- a legacy from my predecessor, the same genius who slung all the ethernet cables hither and thither, put the internal DNS on the corporate web-server (a Win2k box), and basically committed every offense conceivable by someone with my background (software raid? are you kidding me?).

Steve made it clear to me that I had to fix it, and now. It was 10am, and the install was shot -- only later did I realize it was shot in many different ways. I spent the morning snaking cables, tracing power plugs, and moving things around while Greg from the vendor dutifully prepared my two new SAN trays which I had no hope of plugging in.

"The fibre cables are too short," Greg told me at some point. Sure enough, the cables would not reach the head unit several dozen "U"s (units) down. "I'll call Amanda (our faithful implementation manager at the vendor) and order longer cables -- 2 meters?"

"Yeah, that sounds right," I replied absentmindedly while tracing power chords through two racks. The Fucker, my predecessor, had done it again, and stolen power from another rack to provide power to an adjacent rack -- another no-no.

I took a break from tracing power to address a bitchy server with a couple of degraded RAIDs. The salient details are these: Greg, looking at the error messages from the server shook his head several times and said, sympathetically, "no bueno." In order to replace the suspect EIDE cables leading from the source drives to the RAID controllers I had to forcibly remove a row of fans, and in the process warped the fan bracket. When I eventually replaced the cables, the fan bracket was so bent up it would not possibly go back into the chassis, so the fans would have to sit on top of the EIDE cables bracket-less.

"Chris, check this out.." said Greg, from the vendor.

I looked up as he pointed at the second shelf of drives we had just installed.


"We're going to have to send this back."

I looked closer. Greg was pointing to the right corner of the back of the drive shelf, which was clearly bent as if it was dropped. I quickly reviewed in my head receiving the equipment and putting it on a forklift jack -- nope, didn't drop anything.

Let's review:

(1) the power was fucked up, and needed to be fixed.
(2) the fibre cables were too short
(3) one of the trays was damaged

This is truly what it means to be in the weeds, but, it can quickly get worse, and it did just over a week later.

20 June, 2007

White People Suck

Haha: http://www.villagevoice.com/news/0724,tucker,76913,2.html

Is this real, or some surreal parody? You decide:
After more speeches about white flight, white fear, and white power, the meeting adjourned for what Kelso described as "a white man's feast." Single file, the crowd moved down a doubtful flight of stairs into a dank basement. The perimeter of the room was crowded with discarded computers circa 1990, stacked upon a heap of unused old furniture. Two fold-out tables held the repast: hot dogs, hamburgers, slices of American cheese, tubs of pre-prepared macaroni and potato salads, iceberg lettuce, and Lay's potato chips. There was only one kind of bread available: white. Lindstrom stayed at the grill outside, flipping hamburger patties, while everyone else ate dinner inside, away from protesters' cameras.

administrivia: new "datacenter confidential" soon. just let me get through this fucking week in one piece :(

13 June, 2007

Datacenter Confidential #1

The Weeds

If you've read, and you should (even must), Kitchen Confidential, you know a little about being in "the weeds."

If you haven't, shame on you. But being in "the weeds" is something everyone, regardless of their profession, will encounter from time to time. For Tony Bourdain, being In The Weeds may be a night where a line cook called off sick (or in jail, or God knows what else), a Salamander broiler is broken, the dish-washing machine is stuck, and the orders are backing up. Soon, you find yourself so mired in everything going wrong at once that you just can't get your head above water. This is "in the weeds."

Today/yesterday, I was in the shit. Deep.

6am, Tuesday June 12, 2007.

The alarm on my cell phone goes off for the first time. I'm still in a vodka and bad sleep induced fog. I hit snooze.


I'm still not ready to get up, but I know I have an inevitable appointment at 9am with a vendor and I can't reschedule without costing the company I work for an absurd amount of money. I hit snooze again, and move the cell phone closer to my pillow.

This same cycle of snooze hitting and rolling back over continues until about 7:40, when the urge to pee and brush my teeth takes hold. I make myself a cocktail of water, ice and Emergen-C and stand under the shower. Soon it's..


Having wasted this much time just getting out of bed, I call Luxor Cab. If I'm lucky, Christine is working the phones and dispatching, but, as I discover later at the Kilowatt, she has the day off. A vanilla operator answers, one I've talked to dozens of times, and I place my order, suit up, and go outside to wait.


This is a call from DDS. Your cab will arrive in 1 minute..

I hang up and suck on my cigarette with gusto. 30 seconds later, a Luxor van pulls up. Initially I sense some abrasiveness from the cab driver when I tell him to turn right on 17th Street instead of wait for traffic both ways to stop. More pricklies as we make the left onto Vermont. His cell rings, and he has a chipper conversation with the unheard caller about radio stations and the Beatles. Nearing 3rd and Townsend, and ending his call, I sense my "in" and chat him up about radio and the Beatles for the last stretch of the drive. I tip him more than I should, get out, and make a bee line for Border's.


I grab a small bottle of Odwalla orange juice and a large iced Americano. Both of the baristas are cute, and better still cute in a non-white, mixed race kind of way. The cashier has beautiful, full lips.

I take my caffeine and orange-juice breakfast to the tables at the McDonald's across the street, pondering for a split second the merits of getting some sort of food there. As regrettable as McDonald's breakfast is, skipping it on this particular day will turn out to be more regrettable.

The phone rings. Greg from the Vendor is heading across the bridge, he will meet me in 30-45 minutes. I smoke half a cigarette and call my boss, momentarily wondering if he is even up at this hour -- then I remember that Tuesdays are one of his "in office" days. He nags me about prioritizing some queued up jobs for hot-shot clients and I ask him when we will be coming up to the datacenter with a pair of storage-class commodity servers we had delivered last week. 2pm he says. Turns out he's lying to me and actually comes up an hour early, but the stretch between 9am and 1pm is one of the longest I've ever had.


I split McDonald's shortly after talking to the boss man, marched down to the datacenter and snuck in, opened up a couple of racks, moved a few cardboard boxes where machines will be going and plugged in my laptop.

In the datacenter, just like the kitchen, you have to observe and respect mise en place. If I'm a line cook and the datacenter is a service kitchen, my most basic mise en place is in rack A10, where I have a Dell Latitude power brick plugged into an Remote Power Controller (RPC) and a neatly coiled purple Category-5 patch cable plugged into a nearby ethernet switch.

In a perfect world, moreover in a perfect datacenter, patch cables, cross connects and especially power receptacles are neatly ordered. An uptime minded admin will dutifully wire up their machines with an eye toward reliability, usually an acutely obsessive-compulsive eye. Essential servers will have at least two if not more power supplies on alternating 20- or 30-amp power circuits. Critical machines will have expensive duplicate stand-ins, ready to take over in less than a seconds time, with dual-pathed network connections to redundant switches which in turn have redundant power and can take over for each other should one fail. Sleek load-balancers split incoming and outgoing traffic to and from redundantly powered web servers out to the internet at large through a firewall which has a standby peer with which it shares connection state information and finally out a pair of routers connected to the datacenter's upstream customer switches (which are even more sickeningly redundant).

The difference between a perfect world and the world I have inherited will become painfully clear to me very soon.

Greg texts me, he is at the door. I'm game time.


I use my badge to get out of the co-location room. Steve, the facilities manager at the datacenter, sternly warns everyone to do so, because opening the door without badging out will result in a loud alarm and a pissed off facilities manager. I go down the front stairs to let Greg in, pausing first to let out a pair of attractive architects, a man and a woman, who work in the architectural firm which shares space with the datacenter in the building.

The front door is the sole ingress, and there is a proximity panel outside with a keypad that everyone uses to enter -- I use my datacenter badge to open the door day or night, I'm not sure if the architects are badged or have keys to the security dead-bolt lock or have a key-code, but I doubt it is the latter based on a funny anecdote Steve shares with Greg and I as we enter the storage area to retrieve our equipment.

Until recently, I would often enter the building through the back-alley via a keypad-secured door shared by the three tenants of the building: the datacenter, the architects and the telco which occupies most of the first floor. But recently the keypad for that back-alley door was disabled, so Steve tells me, because he discovered that some homeless people had figured out the key-code and were using a storage room to store stolen goods and crack pipes.

I mention that its a wonder that no one had figured out the genius code of "1234" sooner, and Steve nods in agreement.

Time gets funny, but it is still before noon, and I haven't eaten.

We wheel in the 280 or so pounds of hardware into the co-location room and start unpacking. Greg is taking an inventory of the parts while I am grabbing an unused RPC from rack A06 to bolster my receptacle density in A10, where the new stuff is going to get racked up.

I know A10 is where we are going to rack the two new Hitachi Data Systems AMS500 SCSI SAN shelves because (1) its the same rack the head unit is located, (2) it has rack space and (3) I had Steve do a power audit the day before to make sure I had enough amps to power the shelves on the two 20-amp circuits I have in that rack.

It's right about now that my blog using a fixed-width font will come in handy.

Many of you are used to a three-pronged power plug. It looks roughly like this:

| |

In rack A10, I have a vertical power strip with a number of available receptacles just like that. However, the RPC I pull from A06 has a plug that looks like this:

| -

Well, that's fine, because the 20amp receptacles take that sort of plug. I get on my hands and knees (for the first of many times today) and inspect the 20-amp power receptacles provided by the data center. There are two circuits, and each circuit has a single receptacle block with two receptacles each.

To my great disappointment I find both blocks and all four total receptacles occupied. I trace two cords from one circuit to two racked 20-amp RPCs in A10, but the other circuit has two cords that trace to places that mystify me: one goes to a vertical power strip in A10, and another disappears into a hole in the raised floor. Upon further inspection the cord leading to the power strip is seemingly innocuous: the only thing plugged into it is another RPC with 8 receptacles, 6 of which are in use, plugged into various large disk-storage devices. For the cord leading into the floor, I will need to enlist Steve to help me, since there is a special suction tool for lifting floor tiles.

It is here where our day makes a sharp and unfortunate turn for the worse.

04 June, 2007

10 million pounds of sludge from New York and New Jersey

What Seggos discovered—or rediscovered—wasn’t an oil spill, exactly. Rather, it was a mix of gasoline, solvents, and associated poisons bubbling up from the very ground: a thin dribble that betrays the presence of a supertanker’s worth of the stuff submerged in the age-old geology of Greenpoint. It’s actually more than a century’s worth of spills, leaks, and waste dumped by oil companies that has pooled into a vast underground lake, more than 55 acres wide and up to 25 feet thick. First discovered by the U.S. Coast Guard in 1978, the Greenpoint spill has been estimated at anywhere between 17 million and 30 million gallons—three times more oil than the Exxon Valdez spill. That makes it the largest known oil spill in American history.
What is noteworthy about this story, as well as this one[1] is that they are mostly ignored in the mainstream press or written off as leftist or alarmist. Modern "conservatism" (which only seeks to "conserve" power and wealth for those who have obtained a substantial enough amount thereof) does not have any answers, any solutions whatsoever to problems like the one in Brooklyn and the Texas-sized floatilla of plastic and sludge in the Pacific Ocean.

Conservatism's oddly fanatic younger sibling, Libertarianism, does not seek to solve these problems either. Certainly, Ron Paul would not be so presumptuous as to seek lower emissions on cars bought and sold in the US, and would definitely not sanction China, Mexico or India for failing to do the same. Yet, for the amount of waste we produce a year in sheer tonnage, the sheer tonnage of carbon emissions from China blowing over the Pacific Ocean and right into our West Coast is even more troubling.

We are a country in love with the plastic grocery bag. Tens of millions of them end up in the Pacific Ocean, whole or shredded, never to bio-degrade. When the federal government warns pregnant women and young children to avoid certain sea food, this is largely why. This from our own "conservative" government, not Michael Moore or Bill Moyers or even Al Gore.

Well, the time to get alarmed is NOW, people.

[1] Excerpt:
Captain Charles Moore Fate can take strange forms, and so perhaps it does not seem unusual that Captain Charles Moore found his life’s purpose in a nightmare. Unfortunately, he was awake at the time, and 800 miles north of Hawaii in the Pacific Ocean.

It happened on August 3, 1997, a lovely day, at least in the beginning: Sunny. Little wind. Water the color of sapphires. Moore and the crew of Alguita, his 50-foot aluminum-hulled catamaran, sliced through the sea.

Returning to Southern California from Hawaii after a sailing race, Moore had altered Alguita’s course, veering slightly north. He had the time and the curiosity to try a new route, one that would lead the vessel through the eastern corner of a 10-million-square-mile oval known as the North Pacific subtropical gyre. This was an odd stretch of ocean, a place most boats purposely avoided. For one thing, it was becalmed. “The doldrums,” sailors called it, and they steered clear. So did the ocean’s top predators: the tuna, sharks, and other large fish that required livelier waters, flush with prey. The gyre was more like a desert—a slow, deep, clockwise-swirling vortex of air and water caused by a mountain of high-pressure air that lingered above it.

The area’s reputation didn’t deter Moore. He had grown up in Long Beach, 40 miles south of L.A., with the Pacific literally in his front yard, and he possessed an impressive aquatic résumé: deckhand, able seaman, sailor, scuba diver, surfer, and finally captain. Moore had spent countless hours in the ocean, fascinated by its vast trove of secrets and terrors. He’d seen a lot of things out there, things that were glorious and grand; things that were ferocious and humbling. But he had never seen anything nearly as chilling as what lay ahead of him in the gyre.

It began with a line of plastic bags ghosting the surface, followed by an ugly tangle of junk: nets and ropes and bottles, motor-oil jugs and cracked bath toys, a mangled tarp. Tires. A traffic cone. Moore could not believe his eyes. Out here in this desolate place, the water was a stew of plastic crap. It was as though someone had taken the pristine seascape of his youth and swapped it for a landfill.