Datacenter Confidential #1
The Weeds
If you've read, and you should (even must), Kitchen Confidential, you know a little about being in "the weeds."
If you haven't, shame on you. But being in "the weeds" is something everyone, regardless of their profession, will encounter from time to time. For Tony Bourdain, being In The Weeds may be a night where a line cook called off sick (or in jail, or God knows what else), a Salamander broiler is broken, the dish-washing machine is stuck, and the orders are backing up. Soon, you find yourself so mired in everything going wrong at once that you just can't get your head above water. This is "in the weeds."
Today/yesterday, I was in the shit. Deep.
6am, Tuesday June 12, 2007.
The alarm on my cell phone goes off for the first time. I'm still in a vodka and bad sleep induced fog. I hit snooze.
6:08am
I'm still not ready to get up, but I know I have an inevitable appointment at 9am with a vendor and I can't reschedule without costing the company I work for an absurd amount of money. I hit snooze again, and move the cell phone closer to my pillow.
This same cycle of snooze hitting and rolling back over continues until about 7:40, when the urge to pee and brush my teeth takes hold. I make myself a cocktail of water, ice and Emergen-C and stand under the shower. Soon it's..
8:14
Having wasted this much time just getting out of bed, I call Luxor Cab. If I'm lucky, Christine is working the phones and dispatching, but, as I discover later at the Kilowatt, she has the day off. A vanilla operator answers, one I've talked to dozens of times, and I place my order, suit up, and go outside to wait.
8:20
This is a call from DDS. Your cab will arrive in 1 minute..
I hang up and suck on my cigarette with gusto. 30 seconds later, a Luxor van pulls up. Initially I sense some abrasiveness from the cab driver when I tell him to turn right on 17th Street instead of wait for traffic both ways to stop. More pricklies as we make the left onto Vermont. His cell rings, and he has a chipper conversation with the unheard caller about radio stations and the Beatles. Nearing 3rd and Townsend, and ending his call, I sense my "in" and chat him up about radio and the Beatles for the last stretch of the drive. I tip him more than I should, get out, and make a bee line for Border's.
8:27
I grab a small bottle of Odwalla orange juice and a large iced Americano. Both of the baristas are cute, and better still cute in a non-white, mixed race kind of way. The cashier has beautiful, full lips.
I take my caffeine and orange-juice breakfast to the tables at the McDonald's across the street, pondering for a split second the merits of getting some sort of food there. As regrettable as McDonald's breakfast is, skipping it on this particular day will turn out to be more regrettable.
The phone rings. Greg from the Vendor is heading across the bridge, he will meet me in 30-45 minutes. I smoke half a cigarette and call my boss, momentarily wondering if he is even up at this hour -- then I remember that Tuesdays are one of his "in office" days. He nags me about prioritizing some queued up jobs for hot-shot clients and I ask him when we will be coming up to the datacenter with a pair of storage-class commodity servers we had delivered last week. 2pm he says. Turns out he's lying to me and actually comes up an hour early, but the stretch between 9am and 1pm is one of the longest I've ever had.
9:07
I split McDonald's shortly after talking to the boss man, marched down to the datacenter and snuck in, opened up a couple of racks, moved a few cardboard boxes where machines will be going and plugged in my laptop.
In the datacenter, just like the kitchen, you have to observe and respect mise en place. If I'm a line cook and the datacenter is a service kitchen, my most basic mise en place is in rack A10, where I have a Dell Latitude power brick plugged into an Remote Power Controller (RPC) and a neatly coiled purple Category-5 patch cable plugged into a nearby ethernet switch.
In a perfect world, moreover in a perfect datacenter, patch cables, cross connects and especially power receptacles are neatly ordered. An uptime minded admin will dutifully wire up their machines with an eye toward reliability, usually an acutely obsessive-compulsive eye. Essential servers will have at least two if not more power supplies on alternating 20- or 30-amp power circuits. Critical machines will have expensive duplicate stand-ins, ready to take over in less than a seconds time, with dual-pathed network connections to redundant switches which in turn have redundant power and can take over for each other should one fail. Sleek load-balancers split incoming and outgoing traffic to and from redundantly powered web servers out to the internet at large through a firewall which has a standby peer with which it shares connection state information and finally out a pair of routers connected to the datacenter's upstream customer switches (which are even more sickeningly redundant).
The difference between a perfect world and the world I have inherited will become painfully clear to me very soon.
Greg texts me, he is at the door. I'm game time.
9:12
I use my badge to get out of the co-location room. Steve, the facilities manager at the datacenter, sternly warns everyone to do so, because opening the door without badging out will result in a loud alarm and a pissed off facilities manager. I go down the front stairs to let Greg in, pausing first to let out a pair of attractive architects, a man and a woman, who work in the architectural firm which shares space with the datacenter in the building.
The front door is the sole ingress, and there is a proximity panel outside with a keypad that everyone uses to enter -- I use my datacenter badge to open the door day or night, I'm not sure if the architects are badged or have keys to the security dead-bolt lock or have a key-code, but I doubt it is the latter based on a funny anecdote Steve shares with Greg and I as we enter the storage area to retrieve our equipment.
Until recently, I would often enter the building through the back-alley via a keypad-secured door shared by the three tenants of the building: the datacenter, the architects and the telco which occupies most of the first floor. But recently the keypad for that back-alley door was disabled, so Steve tells me, because he discovered that some homeless people had figured out the key-code and were using a storage room to store stolen goods and crack pipes.
I mention that its a wonder that no one had figured out the genius code of "1234" sooner, and Steve nods in agreement.
Time gets funny, but it is still before noon, and I haven't eaten.
We wheel in the 280 or so pounds of hardware into the co-location room and start unpacking. Greg is taking an inventory of the parts while I am grabbing an unused RPC from rack A06 to bolster my receptacle density in A10, where the new stuff is going to get racked up.
I know A10 is where we are going to rack the two new Hitachi Data Systems AMS500 SCSI SAN shelves because (1) its the same rack the head unit is located, (2) it has rack space and (3) I had Steve do a power audit the day before to make sure I had enough amps to power the shelves on the two 20-amp circuits I have in that rack.
It's right about now that my blog using a fixed-width font will come in handy.
Many of you are used to a three-pronged power plug. It looks roughly like this:
In rack A10, I have a vertical power strip with a number of available receptacles just like that. However, the RPC I pull from A06 has a plug that looks like this:
Well, that's fine, because the 20amp receptacles take that sort of plug. I get on my hands and knees (for the first of many times today) and inspect the 20-amp power receptacles provided by the data center. There are two circuits, and each circuit has a single receptacle block with two receptacles each.
To my great disappointment I find both blocks and all four total receptacles occupied. I trace two cords from one circuit to two racked 20-amp RPCs in A10, but the other circuit has two cords that trace to places that mystify me: one goes to a vertical power strip in A10, and another disappears into a hole in the raised floor. Upon further inspection the cord leading to the power strip is seemingly innocuous: the only thing plugged into it is another RPC with 8 receptacles, 6 of which are in use, plugged into various large disk-storage devices. For the cord leading into the floor, I will need to enlist Steve to help me, since there is a special suction tool for lifting floor tiles.
It is here where our day makes a sharp and unfortunate turn for the worse.
If you've read, and you should (even must), Kitchen Confidential, you know a little about being in "the weeds."
If you haven't, shame on you. But being in "the weeds" is something everyone, regardless of their profession, will encounter from time to time. For Tony Bourdain, being In The Weeds may be a night where a line cook called off sick (or in jail, or God knows what else), a Salamander broiler is broken, the dish-washing machine is stuck, and the orders are backing up. Soon, you find yourself so mired in everything going wrong at once that you just can't get your head above water. This is "in the weeds."
Today/yesterday, I was in the shit. Deep.
6am, Tuesday June 12, 2007.
The alarm on my cell phone goes off for the first time. I'm still in a vodka and bad sleep induced fog. I hit snooze.
6:08am
I'm still not ready to get up, but I know I have an inevitable appointment at 9am with a vendor and I can't reschedule without costing the company I work for an absurd amount of money. I hit snooze again, and move the cell phone closer to my pillow.
This same cycle of snooze hitting and rolling back over continues until about 7:40, when the urge to pee and brush my teeth takes hold. I make myself a cocktail of water, ice and Emergen-C and stand under the shower. Soon it's..
8:14
Having wasted this much time just getting out of bed, I call Luxor Cab. If I'm lucky, Christine is working the phones and dispatching, but, as I discover later at the Kilowatt, she has the day off. A vanilla operator answers, one I've talked to dozens of times, and I place my order, suit up, and go outside to wait.
8:20
This is a call from DDS. Your cab will arrive in 1 minute..
I hang up and suck on my cigarette with gusto. 30 seconds later, a Luxor van pulls up. Initially I sense some abrasiveness from the cab driver when I tell him to turn right on 17th Street instead of wait for traffic both ways to stop. More pricklies as we make the left onto Vermont. His cell rings, and he has a chipper conversation with the unheard caller about radio stations and the Beatles. Nearing 3rd and Townsend, and ending his call, I sense my "in" and chat him up about radio and the Beatles for the last stretch of the drive. I tip him more than I should, get out, and make a bee line for Border's.
8:27
I grab a small bottle of Odwalla orange juice and a large iced Americano. Both of the baristas are cute, and better still cute in a non-white, mixed race kind of way. The cashier has beautiful, full lips.
I take my caffeine and orange-juice breakfast to the tables at the McDonald's across the street, pondering for a split second the merits of getting some sort of food there. As regrettable as McDonald's breakfast is, skipping it on this particular day will turn out to be more regrettable.
The phone rings. Greg from the Vendor is heading across the bridge, he will meet me in 30-45 minutes. I smoke half a cigarette and call my boss, momentarily wondering if he is even up at this hour -- then I remember that Tuesdays are one of his "in office" days. He nags me about prioritizing some queued up jobs for hot-shot clients and I ask him when we will be coming up to the datacenter with a pair of storage-class commodity servers we had delivered last week. 2pm he says. Turns out he's lying to me and actually comes up an hour early, but the stretch between 9am and 1pm is one of the longest I've ever had.
9:07
I split McDonald's shortly after talking to the boss man, marched down to the datacenter and snuck in, opened up a couple of racks, moved a few cardboard boxes where machines will be going and plugged in my laptop.
In the datacenter, just like the kitchen, you have to observe and respect mise en place. If I'm a line cook and the datacenter is a service kitchen, my most basic mise en place is in rack A10, where I have a Dell Latitude power brick plugged into an Remote Power Controller (RPC) and a neatly coiled purple Category-5 patch cable plugged into a nearby ethernet switch.
In a perfect world, moreover in a perfect datacenter, patch cables, cross connects and especially power receptacles are neatly ordered. An uptime minded admin will dutifully wire up their machines with an eye toward reliability, usually an acutely obsessive-compulsive eye. Essential servers will have at least two if not more power supplies on alternating 20- or 30-amp power circuits. Critical machines will have expensive duplicate stand-ins, ready to take over in less than a seconds time, with dual-pathed network connections to redundant switches which in turn have redundant power and can take over for each other should one fail. Sleek load-balancers split incoming and outgoing traffic to and from redundantly powered web servers out to the internet at large through a firewall which has a standby peer with which it shares connection state information and finally out a pair of routers connected to the datacenter's upstream customer switches (which are even more sickeningly redundant).
The difference between a perfect world and the world I have inherited will become painfully clear to me very soon.
Greg texts me, he is at the door. I'm game time.
9:12
I use my badge to get out of the co-location room. Steve, the facilities manager at the datacenter, sternly warns everyone to do so, because opening the door without badging out will result in a loud alarm and a pissed off facilities manager. I go down the front stairs to let Greg in, pausing first to let out a pair of attractive architects, a man and a woman, who work in the architectural firm which shares space with the datacenter in the building.
The front door is the sole ingress, and there is a proximity panel outside with a keypad that everyone uses to enter -- I use my datacenter badge to open the door day or night, I'm not sure if the architects are badged or have keys to the security dead-bolt lock or have a key-code, but I doubt it is the latter based on a funny anecdote Steve shares with Greg and I as we enter the storage area to retrieve our equipment.
Until recently, I would often enter the building through the back-alley via a keypad-secured door shared by the three tenants of the building: the datacenter, the architects and the telco which occupies most of the first floor. But recently the keypad for that back-alley door was disabled, so Steve tells me, because he discovered that some homeless people had figured out the key-code and were using a storage room to store stolen goods and crack pipes.
I mention that its a wonder that no one had figured out the genius code of "1234" sooner, and Steve nods in agreement.
Time gets funny, but it is still before noon, and I haven't eaten.
We wheel in the 280 or so pounds of hardware into the co-location room and start unpacking. Greg is taking an inventory of the parts while I am grabbing an unused RPC from rack A06 to bolster my receptacle density in A10, where the new stuff is going to get racked up.
I know A10 is where we are going to rack the two new Hitachi Data Systems AMS500 SCSI SAN shelves because (1) its the same rack the head unit is located, (2) it has rack space and (3) I had Steve do a power audit the day before to make sure I had enough amps to power the shelves on the two 20-amp circuits I have in that rack.
It's right about now that my blog using a fixed-width font will come in handy.
Many of you are used to a three-pronged power plug. It looks roughly like this:
| |
.
In rack A10, I have a vertical power strip with a number of available receptacles just like that. However, the RPC I pull from A06 has a plug that looks like this:
| -
.
Well, that's fine, because the 20amp receptacles take that sort of plug. I get on my hands and knees (for the first of many times today) and inspect the 20-amp power receptacles provided by the data center. There are two circuits, and each circuit has a single receptacle block with two receptacles each.
To my great disappointment I find both blocks and all four total receptacles occupied. I trace two cords from one circuit to two racked 20-amp RPCs in A10, but the other circuit has two cords that trace to places that mystify me: one goes to a vertical power strip in A10, and another disappears into a hole in the raised floor. Upon further inspection the cord leading to the power strip is seemingly innocuous: the only thing plugged into it is another RPC with 8 receptacles, 6 of which are in use, plugged into various large disk-storage devices. For the cord leading into the floor, I will need to enlist Steve to help me, since there is a special suction tool for lifting floor tiles.
It is here where our day makes a sharp and unfortunate turn for the worse.
Comments