The worst first date ever.
I wouldn’t say that I was actively pursuing Keite Davis, but I certainly had been interested in her since we had become friends on a proto-social media site called Friendster.
I think the first thing really caught my attention was her butt.
I doubt that it was a random profile view that prompted me to befriend her years before October 1st, 2009. We probably had friends in common. Her Friendster profile was a photo of her posterior in a well fitted pair of blue lace panties. Her face was also lovely. She was a pixie blonde. Blue eyes and an elven face.
Sometime around 2000 or 2001 we got to chatting, and we continued chatting for years. I had lovers and girlfriends, she had boyfriends, we would bounce our fears and insecurities off one another.
In late summer 2009 she broke up with a boyfriend she had been with for a couple of years and moved to San Francisco. We talked about getting together to meet in person.
On Thursday, the first of October 2009, I was drunk, she was flirting, and I took a leap of faith and called a taxi to go to her apartment.
Two years later I obtained a copy of her death certificate for my folder of “memento mori.”
Earlier that day I attended something called the “Change Advisory Board” or “CAB meeting” at work.
For the young punks of the “continuous integration/continuous deployment” stripe, as well as the lay people out there, a CAB meeting is a meeting of project managers, systems administrators, support technicians and managers. The purpose of the CAB meeting is to review changes to the production service, discuss the potential impact, and vote on whether or not a particular change would be enacted during the two planned maintenance windows: Monday evening or Thursday evening.
Generally, this would entail reviewing code bug fixes to one of the several dozen service components that comprises what we call “The Danger Service”.
That Monday, an upgrade to the Oracle Database software was proposed. We were storing large binary objects in the database: photos, videos. The database performance was taking a hit. In hindsight, the solution would have been to take all large binary objects out of the database and create something like a CDN (content delivery network). I had run into this problem before, but I am not a DBA (database administrator), so I could only offer up anecdotal evidence that what Danger was attempting to do might be a bad idea.
A firmware upgrade to the storage grid that held the database was also proposed. This is a complex system involving hundreds of individual disk drives attached to a million dollars of hardware, connected by another million dollars of fiber-optic cables and switches, connected to millions of dollars of Sun Microsystems hardware.
Keite was a thespian. She had a pure, true and clearly well trained voice. A high, natural soprano. During the brief time Keite and I dated, my roommate, also a singer (albeit not professionally trained) would gently mock Keite for approaching rock standards like “Heartbreaker” by Pat Benetar with too much vibrato.
The database update and storage upgrade were punted to the next CAB meeting, that same Thursday.
Thursday, October 1st, 2009.
Again, other systems administrators and myself objected. Concerns were raised. Microsoft, which had purchased Danger Research Laboratories, Incorporated, had not ponied up money for any kind of “warm backup” for the production database which served millions of end users.
To do so would have cost Microsoft between three- and five-million dollars.
The production database was Oracle. This database ran on Sun Microsystems hardware. The storage array backing this database was on Hitachi Data Systems RAID equipment, connected to the Sun systems over fiber-optic cabling using Brocade switches.
At work we had a regular on-call rotation. Should a problem arise that the technicians in the Network Operating Center (NOC) could not handle on their own, a page message would be sent out to the on-duty systems administrator.
I was not on-call that Thursday.
I was drunk, and getting very intimate with Keite. My Nokia kept ringing. Yes, Nokia. This was an age before the hegemony of the iPhone. Keite was becoming annoyed with the incessant ringing, and demanded that I pick up the phone to “make them go away.”
I was in Keite’s bedroom, on the border of Nob Hill and the Tenderloin, colloquially known as “the tendernob”. My co-worker P was at my apartment, probably having sex with my roommate C. There’s a long story explaining how that came to be which we will get into in some later blog post.
The NOC was calling me, over and over, in desperation. They didn’t want to speak to me per se, but they knew P was seeing my roommate and they needed his expertise.
P was not answering his phone. Why would he? He was not on-call either, and he was "on a date" too.
P was the go-to expert for anything involving large storage arrays. The same arrays described above: hundreds of hard drives, millions of dollars of storage hardware, fiber optic cables, fiber optic switches, and of course host adapters on several million dollars of Sun Microsystems hardware.
The NOC was calling me to get my roommate’s number. Apparently there was some sort of emergency at work.
“The service is down? What do you mean down?”
In spite of the fact that I wasn’t on-call, I went through the outage playbook with the NOC over the phone, pulling my pants on.
“Did you restart the directors? No? The database is offline? Did you restart it? What do you mean the data is gone? Gone gone? Well what did the DBAs say? The data is gone? How the hell is the data gone? The SAN (storage array) is redundant… Ok, ok, I’ll try calling P again…”
I called my roommate and told her that P had better fucking call work.
"Fuck them!" I could hear him in the background.
"C put that fuckhead on the phone, this is serious."
After some cajoling and pleading I finally got P on the phone.
"What do you mean the data's gone?"
"I dunno man, call the NOC."
A couple hours later, P, myself and our other coworker T were on our way to Palo Alto.
Did I mention that it was T's wedding night?
I'll let him write that up in a blog, if he hasn't already.
Meeting up with Keite was exciting for me, it had been a while since I had been with a woman. I went over to her apartment, and we went out to get a drink. Her and I had “known” each other for a decade before meeting face to face. I think she wanted to suss me out first, introduce me to some of her friends, and then maybe if I passed all of the tests, take me to her bed.
Her apartment was tiny, and she had a small mattress on the floor that served as her bed. In my apartment, I had a mattress on a box spring that I got from a former roommate.
That former roommate was a guy we would call Robert Victor Irish, but his real name was Brendan Sheeran. Brendan died in June of 2019.
The data from the database wasn’t completely gone per se, but the indexing system that Oracle used to organize the data stored to disk was complete gibberish. The reason the index was gibberish was because the first 100MB of each of the hundreds of hard drives had been accidentally overwritten during the SAN firmware upgrade.
Myself and several other systems administrators had been urging Microsoft, who had acquired Danger Research Labs for about $400M, to install a warm backup for the production service database. This could have been accomplished for a relatively small investment in infrastructure.
Various lawsuits were filed in the wake of the outage, which lasted more than two weeks. The amount of money Microsoft paid out to settle the lawsuites is not known, but conservative estimates put the amount at around half a billion dollars. Figuring in the money spent during the recovery effort on extra staff, hardware and legal fees, that figure skyrockets to nearly a billion.
Keite was a bit clingy, and I didn’t know how to deal with that. I didn’t have much time at all to deal with Keite after being put on the graveyard shift after the outage, an on-sight systems administrator should the NOC come across some issue they couldn’t deal with.
For the most part, me and several coworkers would sit at our cubes at 3am near the end of one miserable shift after another and chug Maker’s Mark bourbon.
At 4am my shift would end, and I'd kill time before heading to the California Street Station of the local commuter rail Caltrain. The first northbound train came through around 5am. I would pull into SF 4th and King around 6:30am, when most folks were headed to work. I’d go to the news stand to grab a pack of Newports and a handful of airline bottles of Jameson, maybe get an egg sandwich from the coffee shop or, if I was really hurting, Subway.
On a few occasions, early on in our relationship, Keite would stay in my bed and be waiting for me when I got back home, defeated and exhausted.
I don’t normally care to sleep next to someone because the bed gets too hot to sleep comfortably, you can’t sprawl your limbs akimbo, and if you are on the wrong side of the bed, you will have to extricate yourself should you need to go take a leek.
Keite ran cold enough that it wasn’t like sleeping next to a heat sink. She had put on a few pounds since her glory days of offering up pictures of her ass in blue-lace lingerie.
It was nice to put my hand on her smooth, soft belly, in spite of her protests that she was “fat.”
“Baby, you’re not fat. I’m fat. And I love the way your belly feels.”
She would grab my hand and twist around to kiss me on the lips.
BB was a manager at Danger, and he sent out a strongly worded missive in email explaining that each of us on the operations team were to be available 24/7, and failure to answer a call would result in our termination.
I sent a sternly worded letter to BB explaining how my grandmother’s first husband was assassinated right in from my grandmother. A shot in the head, but the papers reported it as a suicide. Grady started out as a strike-breaker before meeting my grandmother Mildred, who was heavily active in the labor movement. When Grady switched sides to side with labor, he was killed for it.
“My grandmother’s first husband lost his life fighting for labor rights and I will be god-damned if you are telling me that I am on-call 24 hours a day in perpetuity and that my job is at stake.”
I forget what I actually wrote, but words to that effect. I sent it to BB, I BCC’d my coworkers, and I copied the VP of HR for Microsoft.
I was instantly a hero among my peers. I was laid off the following March.
Keite and I stopped seeing each other regularly. I had not made enough time for her, and when she did see me, I was exceedingly grumpy and probably self-medicated.
Six months later, I was at her wedding to a really nice guy named Kenny who could devote time and adoration to Keite.
Six months after that, I was at her funeral.
I think the first thing really caught my attention was her butt.
I doubt that it was a random profile view that prompted me to befriend her years before October 1st, 2009. We probably had friends in common. Her Friendster profile was a photo of her posterior in a well fitted pair of blue lace panties. Her face was also lovely. She was a pixie blonde. Blue eyes and an elven face.
Sometime around 2000 or 2001 we got to chatting, and we continued chatting for years. I had lovers and girlfriends, she had boyfriends, we would bounce our fears and insecurities off one another.
In late summer 2009 she broke up with a boyfriend she had been with for a couple of years and moved to San Francisco. We talked about getting together to meet in person.
On Thursday, the first of October 2009, I was drunk, she was flirting, and I took a leap of faith and called a taxi to go to her apartment.
Two years later I obtained a copy of her death certificate for my folder of “memento mori.”
Earlier that day I attended something called the “Change Advisory Board” or “CAB meeting” at work.
For the young punks of the “continuous integration/continuous deployment” stripe, as well as the lay people out there, a CAB meeting is a meeting of project managers, systems administrators, support technicians and managers. The purpose of the CAB meeting is to review changes to the production service, discuss the potential impact, and vote on whether or not a particular change would be enacted during the two planned maintenance windows: Monday evening or Thursday evening.
Generally, this would entail reviewing code bug fixes to one of the several dozen service components that comprises what we call “The Danger Service”.
That Monday, an upgrade to the Oracle Database software was proposed. We were storing large binary objects in the database: photos, videos. The database performance was taking a hit. In hindsight, the solution would have been to take all large binary objects out of the database and create something like a CDN (content delivery network). I had run into this problem before, but I am not a DBA (database administrator), so I could only offer up anecdotal evidence that what Danger was attempting to do might be a bad idea.
A firmware upgrade to the storage grid that held the database was also proposed. This is a complex system involving hundreds of individual disk drives attached to a million dollars of hardware, connected by another million dollars of fiber-optic cables and switches, connected to millions of dollars of Sun Microsystems hardware.
Keite was a thespian. She had a pure, true and clearly well trained voice. A high, natural soprano. During the brief time Keite and I dated, my roommate, also a singer (albeit not professionally trained) would gently mock Keite for approaching rock standards like “Heartbreaker” by Pat Benetar with too much vibrato.
The database update and storage upgrade were punted to the next CAB meeting, that same Thursday.
Thursday, October 1st, 2009.
Again, other systems administrators and myself objected. Concerns were raised. Microsoft, which had purchased Danger Research Laboratories, Incorporated, had not ponied up money for any kind of “warm backup” for the production database which served millions of end users.
To do so would have cost Microsoft between three- and five-million dollars.
The production database was Oracle. This database ran on Sun Microsystems hardware. The storage array backing this database was on Hitachi Data Systems RAID equipment, connected to the Sun systems over fiber-optic cabling using Brocade switches.
At work we had a regular on-call rotation. Should a problem arise that the technicians in the Network Operating Center (NOC) could not handle on their own, a page message would be sent out to the on-duty systems administrator.
I was not on-call that Thursday.
I was drunk, and getting very intimate with Keite. My Nokia kept ringing. Yes, Nokia. This was an age before the hegemony of the iPhone. Keite was becoming annoyed with the incessant ringing, and demanded that I pick up the phone to “make them go away.”
I was in Keite’s bedroom, on the border of Nob Hill and the Tenderloin, colloquially known as “the tendernob”. My co-worker P was at my apartment, probably having sex with my roommate C. There’s a long story explaining how that came to be which we will get into in some later blog post.
The NOC was calling me, over and over, in desperation. They didn’t want to speak to me per se, but they knew P was seeing my roommate and they needed his expertise.
P was not answering his phone. Why would he? He was not on-call either, and he was "on a date" too.
P was the go-to expert for anything involving large storage arrays. The same arrays described above: hundreds of hard drives, millions of dollars of storage hardware, fiber optic cables, fiber optic switches, and of course host adapters on several million dollars of Sun Microsystems hardware.
The NOC was calling me to get my roommate’s number. Apparently there was some sort of emergency at work.
“The service is down? What do you mean down?”
In spite of the fact that I wasn’t on-call, I went through the outage playbook with the NOC over the phone, pulling my pants on.
“Did you restart the directors? No? The database is offline? Did you restart it? What do you mean the data is gone? Gone gone? Well what did the DBAs say? The data is gone? How the hell is the data gone? The SAN (storage array) is redundant… Ok, ok, I’ll try calling P again…”
I called my roommate and told her that P had better fucking call work.
"Fuck them!" I could hear him in the background.
"C put that fuckhead on the phone, this is serious."
After some cajoling and pleading I finally got P on the phone.
"What do you mean the data's gone?"
"I dunno man, call the NOC."
A couple hours later, P, myself and our other coworker T were on our way to Palo Alto.
Did I mention that it was T's wedding night?
I'll let him write that up in a blog, if he hasn't already.
Meeting up with Keite was exciting for me, it had been a while since I had been with a woman. I went over to her apartment, and we went out to get a drink. Her and I had “known” each other for a decade before meeting face to face. I think she wanted to suss me out first, introduce me to some of her friends, and then maybe if I passed all of the tests, take me to her bed.
Her apartment was tiny, and she had a small mattress on the floor that served as her bed. In my apartment, I had a mattress on a box spring that I got from a former roommate.
That former roommate was a guy we would call Robert Victor Irish, but his real name was Brendan Sheeran. Brendan died in June of 2019.
The data from the database wasn’t completely gone per se, but the indexing system that Oracle used to organize the data stored to disk was complete gibberish. The reason the index was gibberish was because the first 100MB of each of the hundreds of hard drives had been accidentally overwritten during the SAN firmware upgrade.
Myself and several other systems administrators had been urging Microsoft, who had acquired Danger Research Labs for about $400M, to install a warm backup for the production service database. This could have been accomplished for a relatively small investment in infrastructure.
Various lawsuits were filed in the wake of the outage, which lasted more than two weeks. The amount of money Microsoft paid out to settle the lawsuites is not known, but conservative estimates put the amount at around half a billion dollars. Figuring in the money spent during the recovery effort on extra staff, hardware and legal fees, that figure skyrockets to nearly a billion.
Keite was a bit clingy, and I didn’t know how to deal with that. I didn’t have much time at all to deal with Keite after being put on the graveyard shift after the outage, an on-sight systems administrator should the NOC come across some issue they couldn’t deal with.
For the most part, me and several coworkers would sit at our cubes at 3am near the end of one miserable shift after another and chug Maker’s Mark bourbon.
At 4am my shift would end, and I'd kill time before heading to the California Street Station of the local commuter rail Caltrain. The first northbound train came through around 5am. I would pull into SF 4th and King around 6:30am, when most folks were headed to work. I’d go to the news stand to grab a pack of Newports and a handful of airline bottles of Jameson, maybe get an egg sandwich from the coffee shop or, if I was really hurting, Subway.
On a few occasions, early on in our relationship, Keite would stay in my bed and be waiting for me when I got back home, defeated and exhausted.
I don’t normally care to sleep next to someone because the bed gets too hot to sleep comfortably, you can’t sprawl your limbs akimbo, and if you are on the wrong side of the bed, you will have to extricate yourself should you need to go take a leek.
Keite ran cold enough that it wasn’t like sleeping next to a heat sink. She had put on a few pounds since her glory days of offering up pictures of her ass in blue-lace lingerie.
It was nice to put my hand on her smooth, soft belly, in spite of her protests that she was “fat.”
“Baby, you’re not fat. I’m fat. And I love the way your belly feels.”
She would grab my hand and twist around to kiss me on the lips.
BB was a manager at Danger, and he sent out a strongly worded missive in email explaining that each of us on the operations team were to be available 24/7, and failure to answer a call would result in our termination.
I sent a sternly worded letter to BB explaining how my grandmother’s first husband was assassinated right in from my grandmother. A shot in the head, but the papers reported it as a suicide. Grady started out as a strike-breaker before meeting my grandmother Mildred, who was heavily active in the labor movement. When Grady switched sides to side with labor, he was killed for it.
“My grandmother’s first husband lost his life fighting for labor rights and I will be god-damned if you are telling me that I am on-call 24 hours a day in perpetuity and that my job is at stake.”
I forget what I actually wrote, but words to that effect. I sent it to BB, I BCC’d my coworkers, and I copied the VP of HR for Microsoft.
I was instantly a hero among my peers. I was laid off the following March.
Keite and I stopped seeing each other regularly. I had not made enough time for her, and when she did see me, I was exceedingly grumpy and probably self-medicated.
Six months later, I was at her wedding to a really nice guy named Kenny who could devote time and adoration to Keite.
Six months after that, I was at her funeral.
Comments