Sysadmin

“Snowden recommends”


Does it strike anyone else odd to see recommendations on how to secure your privacy from someone whose only accomplishment in life was stealing confidential data? It’s a bit like asking a cat how to store tuna; his motives and expertise are not aligned with your interests.

“Raid, raid, go away…”


When a disk fails in a RAID array, the primary risk associated with replacing it is that another disk will fail before the replacement is fully populated. At which point you’ve lost all your data.

So you can understand my concern yesterday morning when, as I was walking into the computer store to buy a replacement SSD for a machine that had failed unexpectedly, I got email from a NAS reporting a failed RAID5 disk, and discovered that I had two servers to fix.

The good news is that the RAID array finished rebuilding successfully while I was rebuilding the server that needed the SSD replaced.

The bad news is that as soon as I finished the long drive home, I got email that the brand-new disk I’d just installed failed. Crib death is possible, but this time the GUI wasn’t responding reliably either, and a root shell on the NAS got hung when I ran dmesg. Which means it was the 5-year-old NAS itself failing, and the disks were probably fine. If I could get them swapped into an identical chassis. That part will have to wait until Tuesday, since while I could buy something today, Amazon Marketplace can’t get me a ReadyNAS Pro 6 on Labor Day.

I’d be more upset if the NFS mounts weren’t still working, allowing me to copy most of the data off to random free space elsewhere. I haven’t quite come up with 8.3TB yet, but a lot of that is archived logs that may have to wait.

Oh, and the original, unrelated SSD replacement? I’m still babysitting that one, too, since the system involved is a fairly gross hack, held together with twist-ties and bubblegum.

My holiday weekend is going just rosy, thanks. How’s yours?

Dear Microsoft,


I think I speak for every network manager and privacy advocate in the world when I say, “fuck you with a rusty crowbar”.

For those who don’t know, one of the features in the Windows 10 beta (and already in the field in Windows Phone 8.1) is WiFi Sense. The short version is, if you share your wireless access with someone, you’re now potentially sharing it with everyone on their contacts list from Outlook.com, Skype, and even Facebook if they link their accounts.

And the network owner can’t stop them from sharing the password, or even find out that it’s happened. MS offers only one way to prevent this from happening, and that’s changing your network’s SSID to contain the string “_optout”. (This article notes that Google has their own magic string to prevent your wireless from being mapped by their cars, so the new hotness is “_optout_nomap”. No doubt Apple will jump on the bandwagon as well, and next year it will have to be “_optout_nomap_nocandyfromstrangers”).

They claim it will only give limited access to all these strangers, and not let them see anything else that’s on your home network, but that requires that we not only believe that there are no security holes in a Microsoft product, but that the raw password is securely stored in three different online services and every stranger’s device.

The only real defense is to use WPA2 Enterprise authentication, which requires a Radius server. Unfortunately, a lot of consumer-grade wireless-only products won’t do that at all. Last time I tried to get a Kindle to use it, it detected it but never actually sent the username/password combination.

[Update: Microsoft’s FAQ for this misfeature includes the statement:

It can take several days for your network to be added to the opted-out list for Wi-Fi Sense. If you want to stop your network from being shared sooner than that, you can change your Wi-Fi network password.

Crowbar, Rusty. Rinse and repeat.]

[Update: just tested a Kindle Paperwhite against a WPA2 Enterprise wireless running TTLS/PAP user-based authentication. It sent an empty password, so no, you can’t protect your home wireless from Wifi Sense if you plan to connect common small devices to it.]

IPSec Rule #1


After years of filling in as Acting Network Guy (now ending, thankfully), I have decided that there’s really only one thing I’m certain of: IPSec problems are always at the other end.

This was demonstrated yet again this morning when we were trying to change our end of a tunnel that had been up for several years from a /32 to a /24, so that additional machines could route through the tunnel. On my end (OpenBSD), this was a one-line change in ipsec.conf and a one-line change in pf.conf. On their end, which involved Real Networking Hardware, it was days of fumbling that left the old /32 tunnel up while they insisted they’d switched their config.

It took a 45-minute conference call this morning to get it straightened out, which I basically spent watching anime with the sound off while their tech guy cleaned cruft out of his configs and rebuilt their end from scratch.

[unrelated, my co-lo had a power outage, and my ancient beta WebEngine never auto-boots completely; you have to hit the big red button on the front. Sadly, the folks at the co-lo had no success with the big red button, so I had to scrounge around the house for the custom console adapter this thing uses, and stop by on my way to work today to watch it fsck the disks. They’ve had several outages this year, and I think it’s time to move the server to one of the statics on my Comcast Business line and then upgrade it to something more powerful than a 500MHz Pentium 3 with 256MB of RAM.]

Dear “funny” employee,


Please do not forge scary emails from HR and accounting on April Fools Day. Nobody’s laughing.

How I feel…


With the number of servers that have caught fire or things that have needed sudden extra attention at work, I ask, in the words of Lyra Lackwit:

"Will things please stop happening now?"

Words that should never appear in the same question:


“auditors” and “Perl script”

Aero considered harmful


Outlook 2013 started breaking for our users last week. Only some of them, and not all at the same time, but the symptom was that the application would no longer start, hanging at the “loading profile…” screen.

The solution is to switch to the “Windows 7 Basic” graphical theme, turning off all the 3D UI decorations.

No, seriously.

And that’s about four days of sysadmin time that we’d like back, please.

“Need a clue, take a clue,
 got a clue, leave a clue”