Sysadmin

Moving Tales


So, we’re in the new building. Well, not the me part of “we”, yet; we’re still down the street from the old place for another month, until that lease is up, allowing them to use our space for storage and staging and such this week. It’ll shave about 15 minutes off my commute when I do get moved there, though, so that’s nice.

We kicked off the move early Thursday morning, powering down the data center and grabbing some essential servers and gear that we wanted back online as soon as they swung over the Cogent line, leaving the rest for the professional server movers (for the first time, this was Not Our Problem).

Anyone in the Bay Area may recall that it started pouring down rain in the wee hours Thursday, the first real rain of the season. Those of us who were still a bit groggy as we finished the server shutdown were suddenly WIDE AWAKE when the fire alarm went off.

…because the rain was coming into the electrical closet through a conduit, right onto the fire control panel. Smaller quantities were also coming into the server room, including a small amount right into the rack where all of the Really Important Servers we were about to hand-carry were located. Fortunately, we got everything out intact.

To our immense surprise, we could plausibly claim to be fully functional this morning when people showed up. They couldn’t all unpack their offices and cubes because things were still being moved and built, but that was also Not Our Problem this time.

Pro tip: when you have to be out of your old building by date X, get the keys to the new one no later than X - 90. Not X - 20ish.

Kyocera printer drivers in El Capitan


So, if you’re trying to add a shiny new office color laser printer (such as the two Kyocera TASKalfa 5052ci that were delivered to our new building), and you’re running Mac OS X El Capitan, and you get a spinning beachball of doom no matter what protocol you try to connect with, here’s what’s going on and how to fix it.

Let’s say you try to use the LPD protocol. As you type each character of the host name, Apple looks it up in DNS and tries to connect via SNMP to figure out what it is. When you click “Add”, it then uses IPP to query for device options.

This is where it goes to hell. The Mac posts a request using HTTP, and the Kyocera says “that shit’s insecure, call me back on HTTPS”. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. Repeat. killall AddPrinter

The same thing happens if you try to use IPP directly, or JetDirect, or pretty much any protocol. Works fine on Sierra or High Sierra, blows chunks on El Capitan.

The only fix is to log into the printer and completely disable SSL. Note that it is not sufficient to simply shut off SSL; you must also disable the “Secure Only” feature for every protocol (and probably login to the printer again, since you’ll be killing the HTTPS page that you’re currently logged in through), or it will keep redirecting you to pages that it knows perfectly well don’t exist.

Not a big fan of shutting off SSL, but redirect-to-broken-SSL is worse.

’nuff said


(original via)

Lesson 22: Blessed Silence


It’s amazing how much less random root and postmaster email I get when 2,000+ servers are down (deliberately, that is; I get a lot more email when they’re down accidentally…).

(they really needed a few more takes for Kate Mulgrew on this scene to make her hand gesture less artificial, but Joel Grey is so perfect that I’m willing to forgive them)

Saturday Update

Doing 85 MPH on the highway with Twinkle Trick blasting on repeat is how I bring over switches from the old building.

With apologies to Total Coelo…


🎶🎶🎶
I work miracles.
Inevitabl-
-y your inept planning makes you need
me work miracles.
🎶🎶🎶

“What do you mean by ‘the data center is on a truck’?”

Things that are not fun, #90


Not fun: starting your second vacation day the same way you started the first one, by connecting to the office and trying to debug a firewall performance issue through a VPN connection that’s affected by it.

Yesterday it mysteriously vanished while we were looking at it, so I didn’t have the opportunity to try a few things. Today, I was able to mitigate the problem by disabling the HFSC queues in PF, reducing the interrupt overhead just enough to compensate for the attack.

The downside to shutting off the throttling is that we risk being DDoS’d by syslog traffic from our products out in the field.

In completely unrelated news, there cannot be any symlinks in the path to a GitLab install, or it goes all wonky.

9/21 Update

So it looks like someone is trying to DDoS our office network. Since the previous attack didn’t keep us offline, they switched to an NTP amplification attack on a machine that had been misconfigured. It was actually kind of pathetic as attacks go; it chewed up some bandwidth (and the incoming packets are still bouncing off my firewall at 1.1 mb/s), but had zero impact on the network.

On contracting…


(items in italics are direct quotes)

He thought we were hiring an architect.

We thought we were hiring a plumber.

J: …and here’s the busted water heater.

C: Yeah, that’s a mess. I’m going to build you a new house.

J: No, we like the house, there’s just some overdue maintenance work that we don’t have time to do ourselves. Small family, really busy.

C: Nobody builds them like this any more. If you want the cool kids to hang out with you, you need the shiny new thing, and that means replacing all of your tools and appliances.

J: This is all name-brand stuff from major vendors, that we have years of experience working with. You’ll be gone in three months, but we’ll be using it for years.

C: embedding into old systems is a) work and b) irrelevant to my career goals

J’s manager’s manager: problem solved.

Dear Atlassian,


Auto-rendering emoticons in a bug-tracker is dumb (because of course someone who pastes 50 lines of log output into a ticket wants to see smiley faces, blue stars, and thumbs-down icons, and SQL always looks better with broken hearts). Repeatedly closing support tickets asking for a way to turn it off is dumber. Forcing your customers to remember to manually comment out this misfeature in an XML file every time they upgrade Jira: priceless.

“Need a clue, take a clue,
 got a clue, leave a clue”