Historians looking back at the history of Europe sometimes wonder at how Europe could collapse from the highly civilized era of the Roman Empire to the primitive era of the Dark Ages. Clearly, marauding Huns, Goths and Vandals had a good deal to do with it, but it still seems a little anomalous that the great engineering skills of the Romans were lost and the Roman roads, aquaducts, baths and sanitation systems fell into disrepair.

The simple truth is that periods of decline happen, and they happen in all walks of life, even without the intervention of marauders.

The IT Dislocation

A period of technical decline happened in computing from 1990 through to, well, just about now. It’s this that explains the apparent anomaly that the IT industry has suddenly rediscovered virtualization, when it’s been hanging around since the middle of the mainframe era, looking for work.

“Hey guys! Look what I’ve invented. It’s kinda round and you can attach it to an axle and move things from here to there with it. It’s gonna be revolutionary as far as transport is concerned. I’m gonna call it a wheel. Good name huh?”

A dislocation occurred in the move from proprietary operating systems that were bound to proprietary hardware (such as OS/400 or Digital’s VMS) to operating systems that were less bound to the hardware; by which I mean Unix, Linux and Windows (only one of them is not proprietary – for 10 bonus points, tell me which).

It’s difficult to point at one specific cause for the decline of the sophisticated proprietary OSes, but the following forces that were in play, all had something to do with it:

1. The emergence and great success of Microsoft, a company run by engineers that had no depth experience of multi-user computing.
2. The business model that Microsoft adopted in its approach to server computing.
3. The triumph of Unix as the basis for OS portability coupled with Unix’s unsophisticated process scheduling capability.
4. The dramatic growth of the computer market causing an escalating skills problem.
5. The realization that Moore’s Law would continue to deliver more power in an exponential fashion.
6. The commoditization of hardware leading to highly inexpensive servers (and desktops).
7. The growth in the number of software vendors writing applications for increase
8. The adoption of Linux and its use on repurposed servers and PCs to implement single applications.

The proprietary OSes were built to run mixed workloads and many applications at the same time. Because they were proprietary they imposed standards for the way that applications should be built and these standards tied the applications to the OS. It was as hard as hell to build portable applications for most of these OSes, so few ever tried.

Unix began to dominate the sever OS market for several reasons. It was written for general portability rather than performance within a specific architecture or specific cpu chip. This meant that it could be used as the basis for any new hardware idea that any vendor developed. Writing a new OS takes a long time; bending Unix took much less time. For that reason IBM, Digital, Sun, HP, Sequent and, just about anyone in the hardware game evolved their own version of Unix.

Unix (in all its versions) was seriously inefficient compared to minicomputers like the DEC VAX, HP 3000 or AS400, but Moore’s Law gradually took care of that problem. Also, Unix didn’t even have a sensible file system, but that proved to be an advantage. The database vendors quickly fell in love with it. If you used Unix, you needed a database.

And that was the beginning of the “march of the servers”; some Windows, some Unix, some Linux. The database servers were followed by email servers and file servers and print servers and DNS servers and web servers and application servers and firewalls and spam filters and appliances, appliances, appliances – stretching from one end of the server farm to another. A server farm was a stud farm. It was a place where servers bred.

The Pendulum Swings Back

By 2005 almost everyone (except the mainframe people, who knew better) had got into the mindset of “a new application = a new server.” Servers were cheap and you could just add them incrementally with new new application. In many areas you didn’t sell software anymore you sold an applianc, and the appliance was just the box that the software came in. It was “plug in and go.”

The reverse swing of the pendulum was caused by:

1. Moore’s Law ran out of steam. Chip vendors responded by putting more cores on the chip, and were hence able to claim that the newer chips were more powerful (as indeed they were).
2. The server farms began to run out of space and power, and ran into cooling problems too. When you realize that computer room space is the most expensive office space there is, you understand why organizations don’t particularly like server farms.
3. The resource efficiency of the individual servers started to become a matter of record as soon as companies launched server consolidation efforts. The resource inefficiency had become almost scandalous. In some Unix environments I came across the average resource utilization was 6% and in one Windows server farm I stumbled into it was only 4%. Imagine buying 20 room house and only living in one room.

So that is the impulse to virtualization. That is: “why virtualization, and why now?”

This is a posting in the Virtualization Focus Series. Click here to see an index of such postings.