07 November, 2007

Datacenter Confidential #11

Shotgun Re-org

A shotgun re-org is my own term for what happens when a business runs out of operating funds and immediately shuts down operations, or worse, posts a message like this:

Feedster is Changing...
We'll be back up soon with exciting news!
I am citing this particular (and very recent) example by name because they failed to notify my company, their paying client, of their feed shutdown.

As a result, my pager would go off twice every six hours: first, when my cronjob would complain that no new feeds were coming in and second, when our offshore "noc" would ditto the page (I swear, I think they just exist to infuriate me).

Last night, as I was going to bed, I had the amusing thought that "wouldn't it be funny if they had gone tits up?" This morning, that intuition was confirmed when my co-worker IMed me to tell me that they had, indeed, gone tits up. I literally LOLed, and I'm not ashamed to admit it.

I worked for a marketing firm in 1997 that had a version of a shotgun re-org. What had happened is that they bounced a couple of payroll checks, and after that I left. Not long after that, a similar "Coming soon!.." not appeared on their main page, I can only assume there was a mass quit or mass firing. Amazingly, years later I checked their web page -- somehow they got additional funding and actually pulled themselves out of the weeds long enough to be acquired by a larger firm.

Conversely, I have been on the ass-end of shotgun re-orgs as an employee of an acquisition-oriented company that grabs a company only to have all of its employees quit en-masse. In one case, myself and ravinald worked practically around the clock for three or four days to integrate user shells, dialup accounts, DNS, dedicated customers and mail from an acquisition into my network.

Regarding the latter service, mail, when you consider that in 1998 one couldn't simply turn to Google to solve all your problems, it is nothing short of miraculous that we were able to convert and import this companies qmail/maildir spools into a Berkeley mail spool format under sendmail replete with aliases and virtual domains, given the extreme conditions: neither myself nor ravi had barely heard of qmail and maildir, much less used it; the large number of users; the, for the time, size of the dataset (we enforced mail quotas, they did not), number of files, NIC wire speeds (we ended up moving their mail server onto the same switch as ours to speed up xfers) and the method we used to concat the files together (formail and procmail).

This was, of course, back when you actually had to read man files. I oftentimes miss those days.

Coming soon.. trading porn with Cisco support at 4am!

No comments: