Sysadmin

On contracting...


(items in italics are direct quotes)

He thought we were hiring an architect.

We thought we were hiring a plumber.

J: …and here’s the busted water heater.

C: Yeah, that’s a mess. I’m going to build you a new house.

J: No, we like the house, there’s just some overdue maintenance work that we don’t have time to do ourselves. Small family, really busy.

C: Nobody builds them like this any more. If you want the cool kids to hang out with you, you need the shiny new thing, and that means replacing all of your tools and appliances.

J: This is all name-brand stuff from major vendors, that we have years of experience working with. You’ll be gone in three months, but we’ll be using it for years.

C: embedding into old systems is a) work and b) irrelevant to my career goals

J’s manager’s manager: problem solved.

Dear Atlassian,


Auto-rendering emoticons in a bug-tracker is dumb (because of course someone who pastes 50 lines of log output into a ticket wants to see smiley faces, blue stars, and thumbs-down icons, and SQL always looks better with broken hearts). Repeatedly closing support tickets asking for a way to turn it off is dumber. Forcing your customers to remember to manually comment out this misfeature in an XML file every time they upgrade Jira: priceless.

"Fun" with Jira


Twelve days after moving the company from Bugzilla to Jira, it looks like we’re mostly out of the woods.

Except for people whose workflow is built around archived links to Bugzilla URLs, and still try to use it even though I’ve made it read-only, moved it to a different URL, and replaced the old URL with a list of links mapping old bug numbers to their new issue IDs.

And people who crash the Jira server and take down corporate email by using the bulk-update feature to rearrange hundreds of imported bugs at the same time on their own initiative, without disabling notifications.

Bulk updates are now restricted to three users, not including the admin group. Naturally, the person who sent 8,000 emails on Monday abused admin privileges to add himself to the new bulk-update group and send 4,000 more this morning, costing him his admin privileges (and his consciousness, if I could have reached through the Internet and throttled him). Perhaps another day I’ll do an “ambitious Russian developer” rant, about the sort of contractors who would rather break into a server to restart a daemon than send email to IT.

The #1 problem with Jira is that it sends out notifications every time a user picks his nose, so even when people aren’t abusing the bulk-update feature, users are complaining about getting five notifications when someone edits an issue.

My response is to say, “this is what you asked for”.

Reasons to have an OpenBSD router at home, Amazon Wand Edition


Since the new Amazon Dash Wand is effectively free for Prime customers, and it gives you a home-automation controller, bar-code scanner, and a hand-held Alexa device that is not always listening, I ordered one.

When it arrived this morning, I followed the instructions, opened the Amazon app on my iPhone, and went through the setup process. Wifi Fail. Wifi Fail. Wifi Fail. “You should contact customer service”.

The first 20+-minute call went through a bunch of cookbook questions about who my Internet provider was, and how to change the channel on my router. I had a brief flashback to the Seventies, then realized their script assumed Comcast meant “all-in-one cable modem, router, and wireless access point”. I played along, knowing this would make no difference, and the call eventually ended in an RMA.

I was curious to see if it really was a wireless problem, so I logged into the OpenBSD router, checked the DHCP logs, and found an entry for a new Amazon MAC address. I fired up tcpdump and went through the setup again, and sure enough, the device got DHCP, connected to the Internet for DNS, connected to an Amazon server, and then started trying to talk to a public (non-Amazon) NTP server to set its date and time.

It failed every time. Annoyingly, it wasn’t even looking in DNS for its NTP server; the addresses were hardcoded in either the build or the config it had downloaded.

So, armed with the knowledge that the hardware was fine, I tried to get back through to customer service with this knowledge. An hour later, after two different people tried to debug phone app, wireless and bluetooth problems (including telling me to turn on GPS on my phone!), I finally got someone to twiddle the right bits so it could connect to servers that were up, and then cancel the RMA.

Now I have a Dash Wand. Ho, ho, ho.

"Goodnight, Bugzilla"


Just shut down our Bugzilla server as Our Little Elves start the task of migrating all the data into Jira. Which would have been done six weeks ago if Someone hadn’t demanded at the last minute that we rearrange the product structure to reflect Our New Way, and then spent a month being too busy to actually reveal the new design. And then IT still had to do all the work of creating the new products and components, and actually updating the bug DB.

(in fairness, I volunteered for that last bit, because I knew exactly how to do it; Someone (yes, The Same Someone) had asked me to do it at Digeo, over ten years ago…)

Meanwhile, the team responsible for designing the Jira ticket workflow finally delivered a picture of it at 7:55pm. Our Little Elves were scheduled to start the migration at 8pm.

I have a hunch there’s going to be widespread panic on Monday, as they get what they asked for, good and hard.

“Dear Bugzilla, please take us back. We’ll never cheat on you again.”

Got Backups?


(via)

Definition of a good time


You’ve spent the past two weeks being yelled at by a user for not getting their external partner’s incoming connection to work

and you’ve had a tcpdump running for an entire week showing that no connection attempts have been made from the IP addresses the partner provided

and they schedule a conference call at a time that’s convenient for the partner’s third-world contractors

and they confirm their IP addresses in the chat but the test fails again

and your tcpdump shows them coming in from a completely different IP address

and they start to wrap up the meeting saying they’ll contact their network team who hadn’t been invited and reschedule for the next day

and you have to yell into the microphone to tell them to try again right now since you’ve just added their real IP address to the firewall

and they confirm that it works but continue talking about who’s going to do what and how they will communicate the results and who will be responsible for the next step and oh fuck who cares you stopped listening two minutes ago

and you close the multiple tickets created by the user who doesn’t understand that CC’ing the helpdesk on every email keeps creating new tickets

and the partner emails a list of 26 possible IP addresses that does not include the two they originally claimed were the only ones they use

and then they try to schedule another meeting anyway and you reject the invite twice

and you go back to bed.

…and reach for earplugs because the neighbor puts his dog out when he goes to work and it barks and whines all day long and sounds remarkably like one of your users.

Retirement Party!


Not mine, sadly, but the ancient NetEngine WebEngine that was dotclue.org for so many years. I pulled it from the co-lo on my way into work this morning, and its reward for fourteen years of faithful service will be a disk scrub and an e-waste bin.

By the way, for all the sometimes-deserved criticism that OpenBSD and its wranglers get, I was still running v3.3 without anyone ever successfully breaking in. I locked it down with a very small set of services, and required non-root logins with ssh keys, and Theo’s Paranoid Army took care of the rest. I applied the various security patches that came out in 2003-2004, but that’s it.

I don’t recommend not updating your server for 14 years, but you can go a lot longer between updates if you start with something designed for security.

Amusingly, I still own the even-older server that hosted munitions.com back in the days when it was shared between folks at WebTV, but I doubt I have anything left that could mount those disks to scrub them, so they’ll just get the sledgehammer treatment, and then go into the e-waste bin.

“Need a clue, take a clue,
 got a clue, leave a clue”