Sysadmin

With apologies to Total Coelo…


🎶🎶🎶
I work miracles.
Inevitabl-
-y your inept planning makes you need
me work miracles.
🎶🎶🎶

“What do you mean by ‘the data center is on a truck’?”

Things that are not fun, #90


Not fun: starting your second vacation day the same way you started the first one, by connecting to the office and trying to debug a firewall performance issue through a VPN connection that’s affected by it.

Yesterday it mysteriously vanished while we were looking at it, so I didn’t have the opportunity to try a few things. Today, I was able to mitigate the problem by disabling the HFSC queues in PF, reducing the interrupt overhead just enough to compensate for the attack.

The downside to shutting off the throttling is that we risk being DDoS’d by syslog traffic from our products out in the field.

In completely unrelated news, there cannot be any symlinks in the path to a GitLab install, or it goes all wonky.

9/21 Update

So it looks like someone is trying to DDoS our office network. Since the previous attack didn’t keep us offline, they switched to an NTP amplification attack on a machine that had been misconfigured. It was actually kind of pathetic as attacks go; it chewed up some bandwidth (and the incoming packets are still bouncing off my firewall at 1.1 mb/s), but had zero impact on the network.

On contracting…


(items in italics are direct quotes)

He thought we were hiring an architect.

We thought we were hiring a plumber.

J: …and here’s the busted water heater.

C: Yeah, that’s a mess. I’m going to build you a new house.

J: No, we like the house, there’s just some overdue maintenance work that we don’t have time to do ourselves. Small family, really busy.

C: Nobody builds them like this any more. If you want the cool kids to hang out with you, you need the shiny new thing, and that means replacing all of your tools and appliances.

J: This is all name-brand stuff from major vendors, that we have years of experience working with. You’ll be gone in three months, but we’ll be using it for years.

C: embedding into old systems is a) work and b) irrelevant to my career goals

J’s manager’s manager: problem solved.

Dear Atlassian,


Auto-rendering emoticons in a bug-tracker is dumb (because of course someone who pastes 50 lines of log output into a ticket wants to see smiley faces, blue stars, and thumbs-down icons, and SQL always looks better with broken hearts). Repeatedly closing support tickets asking for a way to turn it off is dumber. Forcing your customers to remember to manually comment out this misfeature in an XML file every time they upgrade Jira: priceless.

“Fun” with Jira


Twelve days after moving the company from Bugzilla to Jira, it looks like we’re mostly out of the woods.

Except for people whose workflow is built around archived links to Bugzilla URLs, and still try to use it even though I’ve made it read-only, moved it to a different URL, and replaced the old URL with a list of links mapping old bug numbers to their new issue IDs.

And people who crash the Jira server and take down corporate email by using the bulk-update feature to rearrange hundreds of imported bugs at the same time on their own initiative, without disabling notifications.

Bulk updates are now restricted to three users, not including the admin group. Naturally, the person who sent 8,000 emails on Monday abused admin privileges to add himself to the new bulk-update group and send 4,000 more this morning, costing him his admin privileges (and his consciousness, if I could have reached through the Internet and throttled him). Perhaps another day I’ll do an “ambitious Russian developer” rant, about the sort of contractors who would rather break into a server to restart a daemon than send email to IT.

The #1 problem with Jira is that it sends out notifications every time a user picks his nose, so even when people aren’t abusing the bulk-update feature, users are complaining about getting five notifications when someone edits an issue.

My response is to say, “this is what you asked for”.

Reasons to have an OpenBSD router at home, Amazon Wand Edition


Since the new Amazon Dash Wand is effectively free for Prime customers, and it gives you a home-automation controller, bar-code scanner, and a hand-held Alexa device that is not always listening, I ordered one.

When it arrived this morning, I followed the instructions, opened the Amazon app on my iPhone, and went through the setup process. Wifi Fail. Wifi Fail. Wifi Fail. “You should contact customer service”.

The first 20+-minute call went through a bunch of cookbook questions about who my Internet provider was, and how to change the channel on my router. I had a brief flashback to the Seventies, then realized their script assumed Comcast meant “all-in-one cable modem, router, and wireless access point”. I played along, knowing this would make no difference, and the call eventually ended in an RMA.

I was curious to see if it really was a wireless problem, so I logged into the OpenBSD router, checked the DHCP logs, and found an entry for a new Amazon MAC address. I fired up tcpdump and went through the setup again, and sure enough, the device got DHCP, connected to the Internet for DNS, connected to an Amazon server, and then started trying to talk to a public (non-Amazon) NTP server to set its date and time.

It failed every time. Annoyingly, it wasn’t even looking in DNS for its NTP server; the addresses were hardcoded in either the build or the config it had downloaded.

So, armed with the knowledge that the hardware was fine, I tried to get back through to customer service with this knowledge. An hour later, after two different people tried to debug phone app, wireless and bluetooth problems (including telling me to turn on GPS on my phone!), I finally got someone to twiddle the right bits so it could connect to servers that were up, and then cancel the RMA.

Now I have a Dash Wand. Ho, ho, ho.

“Goodnight, Bugzilla”


Just shut down our Bugzilla server as Our Little Elves start the task of migrating all the data into Jira. Which would have been done six weeks ago if Someone hadn’t demanded at the last minute that we rearrange the product structure to reflect Our New Way, and then spent a month being too busy to actually reveal the new design. And then IT still had to do all the work of creating the new products and components, and actually updating the bug DB.

(in fairness, I volunteered for that last bit, because I knew exactly how to do it; Someone (yes, The Same Someone) had asked me to do it at Digeo, over ten years ago…)

Meanwhile, the team responsible for designing the Jira ticket workflow finally delivered a picture of it at 7:55pm. Our Little Elves were scheduled to start the migration at 8pm.

I have a hunch there’s going to be widespread panic on Monday, as they get what they asked for, good and hard.

“Dear Bugzilla, please take us back. We’ll never cheat on you again.”

Got Backups?


(via)