Sysadmin
Why I Love Users, a parable of hacking
(based on a true story from my OSU-CIS days…)
User A notices that the department has installed a new sprinkler system. He immediately proceeds to find out everything about how it works, what it can do, and how reliable it is. People are astonished at how much he knows about it, and he basks in the warm glow of praise. One day, he uncovers a serious implementation flaw that no one knows about, and makes veiled references to it for several months, never to the people who are in a position to fix it. Finally, he decides to show people how bad the system is, and sets fire to the building. He’s careful to make sure that no one gets hurt, and that the damage is minor. When the fire-fighters approach him with blood in their eyes and axes in their hands, he smiles quietly and says, “I told you so; you should have listened.”
This being just a story, I feel compelled to permit the fire-fighters to hack the little toad to pieces, shouting “LIKE HELL YOU DID!”
The moral of this story is a variation on the Golden Rule:
"Do unto others as you would have them do unto you, because they can do unto you a lot harder."
Tales From The Help Desk
E-mail exchange between user and sysadmin at OSU-CIS, long ago and far away…
User: I was wondering how to send mail to someone on the VAX systems.
Sysadmin: Which ones?
U: It’s the VAX 386 systems. I know the three unique letters to identify this person. Thanks.
S: That doesn’t help. Perhaps I should instead have asked whose VAX systems.
U: It is the VAX at BF Goodrich in Avon Lake, OH. Hope this helps.
S: (crycrycry)
Sanitizing Apache with PF
About 45 minutes elapsed between the moment that I first turned this server on and the arrival of the first virus/worm/hacker probes. It was obvious that most of them were looking for Windows-based web servers, so they were harmless to me.
Still, I like to review the logs occasionally, and the sheer volume of this crap was getting annoying. Later, when I raised munitions.com from the dead, I discovered that it was getting more than 30,000 hits a day for a file containing the word “ok”. Worst of all, as I prepare to restore my photo archives, I know that I can’t afford to pay for the bandwidth while they’re slurped up by every search engine, cache site, obsessive collector, Usenet reposter, and eBay scammer on the planet.
Enter PF, the OpenBSD packet filter.
Unrepentant sinner
Undergrads love free Internet porn. This is not news. Undergrads will go to great lengths to hide their porn collections from the sysadmins. This also is not news. Sometimes they outsmart themselves. This is just plain fun.
Sometimes it's not the network
My job was Unix support for Corporate Services, which basically referred to everything in the company that wasn’t related to developing, selling, or training customers how to use our products. In practice, though, it usually just meant MIS, because HR and Legal were composed entirely of Mac people, who had their own support team.
The oddest exception started one day when an HR manager asked me to help him set up a beta-test of a Lotus Notes-based applicant tracking system. The application was being developed on OS/2 servers and PC clients, but we wanted to test it with a SunOS server and Mac clients, since that’s what we had.
Backups? What backups?
A.J. was worried. For several months, he’d been growing more and more concerned about the reliability of the Unix server backup system that he operated every day. He was just the latest in a long string of junior contractors paid to change tapes, but he actually cared about doing a good job, and something wasn’t right.
He had raised his concerns with the manager of Core Services and the Senior System Administrators who were responsible for the corporate infrastructure, but they assured him that any problems were only temporary, and that he should wait until they had the new system in place. A.J. resigned himself to pretending to do his job, and grudgingly agreed to stall for more time whenever a restore was requested that he couldn’t accomplish.
And then the system just stopped working.
How not to write a job description...
…1995 Edition. This is what happens when your senior sysadmin leaves, and there’s no one left with even a tiny grasp of what the job involves. It happened to OSU-CIS; don’t let it happen to you!
Spindles and Platters and Heads, oh my!
There’s a story I love to tell, a cautionary tale about an incompetent manager, his ass-covering sysadmins, and the company that they could have destroyed together. At some point I’ll write it up here, but the short version goes like this: “two-thirds of the file servers hadn’t been backed up in six months, and they knew this.”
