Once every three months, we sent the whole company home while we tore the computer room apart and did all sorts of maintenance work. During my first quarterly downtime, the top item on my list was installing a new BOSS controller into the Solbourne that was our primary Oracle database server. Like any good database, it needed an occasional disk infusion to keep it happy, and there was no room on the existing SCSI controllers.
So I had a disk tray, a bunch of shiny new disks, a controller card, and media to upgrade the OS with. The BOSS was only supported in the latest version, and this being the server that kept the books, it was upgraded only when necessary.
I’ve been working with Perl since about two weeks before version 2.0 was released. Over those fifteen years, I’ve seen a lot of hairy Perl scripts, many of them mine.
None of them can compare to the monster that lurks in the depths of our service, though. Over 8,000 lines of Perl plus an 8,000-line C++ module, written in a style that’s allegedly Object Oriented, but which I would describe as Obscenely Obfuscated (“Hi, Andrew!”).
We have five large servers devoted to running it. Each contributes three CPUs, three gigabytes of memory, and 25 hours of runtime to the task (independently; we need the redundancy if one of them crashes). Five years ago, I swore a mighty oath to never, ever get involved with the damned thing.
Then it broke. In a way that involved tens of thousands of unhappy customers.
(based on a true story from my OSU-CIS days…)
User A notices that the department has installed a new sprinkler system. He immediately proceeds to find out everything about how it works, what it can do, and how reliable it is. People are astonished at how much he knows about it, and he basks in the warm glow of praise. One day, he uncovers a serious implementation flaw that no one knows about, and makes veiled references to it for several months, never to the people who are in a position to fix it. Finally, he decides to show people how bad the system is, and sets fire to the building. He’s careful to make sure that no one gets hurt, and that the damage is minor. When the fire-fighters approach him with blood in their eyes and axes in their hands, he smiles quietly and says, “I told you so; you should have listened.”
This being just a story, I feel compelled to permit the fire-fighters to hack the little toad to pieces, shouting “LIKE HELL YOU DID!”
The moral of this story is a variation on the Golden Rule:
"Do unto others as you would have them do unto you, because they can do unto you a lot harder."
E-mail exchange between user and sysadmin at OSU-CIS, long ago and far away…
User: I was wondering how to send mail to someone on the VAX systems.
Sysadmin: Which ones?
U: It’s the VAX 386 systems. I know the three unique letters to identify this person. Thanks.
S: That doesn’t help. Perhaps I should instead have asked whose VAX systems.
U: It is the VAX at BF Goodrich in Avon Lake, OH. Hope this helps.
About 45 minutes elapsed between the moment that I first turned this server on and the arrival of the first virus/worm/hacker probes. It was obvious that most of them were looking for Windows-based web servers, so they were harmless to me.
Still, I like to review the logs occasionally, and the sheer volume of this crap was getting annoying. Later, when I raised munitions.com from the dead, I discovered that it was getting more than 30,000 hits a day for a file containing the word “ok”. Worst of all, as I prepare to restore my photo archives, I know that I can’t afford to pay for the bandwidth while they’re slurped up by every search engine, cache site, obsessive collector, Usenet reposter, and eBay scammer on the planet.
Enter PF, the OpenBSD packet filter.
Undergrads love free Internet porn. This is not news. Undergrads will go to great lengths to hide their porn collections from the sysadmins. This also is not news. Sometimes they outsmart themselves. This is just plain fun.
My job was Unix support for Corporate Services, which basically referred to everything in the company that wasn’t related to developing, selling, or training customers how to use our products. In practice, though, it usually just meant MIS, because HR and Legal were composed entirely of Mac people, who had their own support team.
The oddest exception started one day when an HR manager asked me to help him set up a beta-test of a Lotus Notes-based applicant tracking system. The application was being developed on OS/2 servers and PC clients, but we wanted to test it with a SunOS server and Mac clients, since that’s what we had.
A.J. was worried. For several months, he’d been growing more and more concerned about the reliability of the Unix server backup system that he operated every day. He was just the latest in a long string of junior contractors paid to change tapes, but he actually cared about doing a good job, and something wasn’t right.
He had raised his concerns with the manager of Core Services and the Senior System Administrators who were responsible for the corporate infrastructure, but they assured him that any problems were only temporary, and that he should wait until they had the new system in place. A.J. resigned himself to pretending to do his job, and grudgingly agreed to stall for more time whenever a restore was requested that he couldn’t accomplish.
And then the system just stopped working.