21 October, 2011

DevOps: the jig is up

I will be the first to admit that systems administration is a protection racket: pay us, or something *bad* will happen to your network.

Of course, the fact that bad things do happen to servers and networks -- worse, if you host a site on the internet you invite all sorts of badness from bored teenagers to Chinese government-funded hacker farms -- keeps us systems administrators out of jail for extorting money (our salaries) from companies for protection.

As if that's not bad enough, us systems administrators are notorious for erecting barriers between developers and their masters, dumb management, and critical production systems that the systems administrators are tasked with maintaining. QA, regression testing, staging and peer review are just some of the tools in the "stymy development" tool box systems administrators use to slow down the development cycle from "pretty fast" to "reckless, stupid abandon".

Developers, who know in their hearts that they can do no wrong, and managers, whose lack of technical knowledge and thirst for capital knows no bounds, have teamed up to do an end-run around those pesky systems administrators. The key to their success in this noble goal lays in two buzzwords you are likely to hear being tossed around: "the cloud" and "devops".

"The cloud" is a magical place where you can, for a price, deploy as many crappy virtual Linux instances as you desire on demand without having to directly pay someone who knows what they are doing.

Are you fresh out of college with only a glancing familiarity of Linux? Can you point and click? Congratulations, you are qualified to become a "cloud" administrator. Let's deploy some LAMP servers!

Do you value high availability, scrutinize resource usage and have a mind towards stability and security? Go away, dinosaur, you must be a sysadmin.

"DevOps" is really just development, sans quality control, regression testing or change management. A sysadmin would not fare well in this environment, wanting to focus on operational concerns instead of pumping out redundant, useless and hastily written perl or PHP code 10 hours a day, 6 days a week. Those lazy sysadmins will try to limit their time in the office to something under 50 hours, and they will try to excuse their lack of ambition and team spirit on things like "being on call" or "having a life". First, "the cloud" means you never have to be on-call, because how could "the cloud" ever fail? How? Stop talking.. Second, it's your "life" or our founder's million dollar payout when Zynga buys us. Pick a side (hint: the one that doesn't involve getting some 28 year old fuck-head "CTO" rich from a Zynga stock trade means you're fired).

The cloud is a good way to quickly develop and deploy a new product, but should not be the be-all and end-all of internet or mobile app development; despite it's ease of use those without knowledge of operations best practices often make "rookie" errors in designing and deploying their applications. And while an agile development cycle can encourage innovation, doing so without the safety harness of testing and change management under the watchful gaze of experienced systems administrators invites disaster -- especially if your development cycle is dev straight to prod.

25 July, 2011


Some of these posts are delightfully unhinged. Ah, drugs.

Working on other projects for a while, we will return to Goofballs in the future.

12 May, 2011

Editing a letter to Dad about what it is I do..

The three laws of Arthur C. Clarke:

  • When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

  • The only way of discovering the limits of the possible is to venture a little way past them into the impossible.

  • Any sufficiently advanced technology is indistinguishable from magic.

The third law applies best to how most people perceive what I do for a living. I will try to dispel magic in lieu of science, and I will try to do so semi-chronologically.

In the early 1980s I had been given a number of Apple computers which were either on loan from or purloined from the law offices of my father; some he bought, outright, seeing my interest in computers. Nevertheless my very first command-line interpreter (CLI) was the built-in Apple BASIC interpreter that all Apples came with, on 5.25” floppy disks.

A typical program might look something like this:


20 PRINT CHR$(7)

30 GOTO 10

This program would endlessly beep until you cancelled it with a control-C (I had not yet learned about trapping system control variables, but I soon would). Between the ages of about 5 to around 9 years old, the Apple, it's BASIC and some of it's MC6501/6502/65C021 assembly language.

Around the time I turned 10 I got a brand new Mac Plus, which boasted an entire 1MB of RAM. My precious command-line suddenly disappeared and was replaced with a visual interface (GUI – graphical user interface). I hated it.

This new technology wasn't magical to me, it was annoying. An annoying obstacle. So I researched it, learned what I could about it.

I learned that the filesystem kept files organized as pure data files and files that could be run, and the files that could be run had a “fork” called the resource fork. I learned that Mac System OS had a utility to manipulate that called ResEdit2. ResEdit opened up a whole new world in the Macintosh operating system to me, but that wasn't the end of my exploration.

Soon I discovered MACSBug3: a low-level debugger from Motorola, makers of the Macintosh CPU the 68k or MC68000 series of processors. The Mac Plus ran on an MC68000, the Mac SE also on the same 68k chip but with wider “motherboard bandwidth” (better access to the 1MB of RAM, or, 2MB), the Mac SE ran on the MC68010. The Mac LC ran on the MC68020, and unsurprisingly the LC-30 ran on the MC68030 (same as the Mac SE-30).

I think it was the Mac SE30 that we had which had this beast attached:

Pictured above is a Jasmine 20MB SCSI Hard Disk Drive. In 1988, this cost $700. Now, my network interface card probably has more on-board cache. A single gigabyte would run into the tens of thousands.

Exploring the Jasmine drive got me into quite a bit of trouble: for some reason, I thought it would be fun to put a “password” on the drive so I could keep my stuff private – not an unexpected impulse for a precocious 12 year old; but not good for a 40 lawyer with data on that drive. Worse, still, when I forgot the password.

And in that regard, my father was my first partner in hacking something of value: we engaged the makers of the equipment, they directed us to a piece of software called “SCSIEditor” which I continued to use for many years. The software allowed you to edit the boot blocks of a SCSI block device, erasing the password. This was my first experience with sector editing on a low-level block device, but it would not be my last.

1The Motorola chip in the Apple ][, ][e and ][c respectively.