Sysadmin

Well, that was special…


I use Amazon’s RedHat-based Linux distribution to run this site in their cloud, with Nginx as the main web server, and Lighttpd for CGI-ish things that are reverse-proxied by Nginx.

Amazon’s been pretty good at maintaining the RPMs, to the point that I don’t worry much about running “yum update” and rebooting at frequent intervals, although I do update my test instance before the real one.

So it was not pleasant to go through a typical update, surf to my site, and find the Lighttpd default page instead of my blog. Whoever packaged up the latest release had it overwrite /etc/rc.d/init.d/lighttpd, blowing away my configuration and replacing it with the default one. And it started up before Nginx, so it claimed the ports.

(and before you ask, I would have put my customizations in /etc/sysconfig/lighttpd if the script had been written in a way that allowed those particular changes; the workaround is going to be to copy it to a new name and disable the original)

Fortunately I keep all configs under source control, so I simply reverted that file and restarted everything, but it’s still annoying.

Unrelated, tomorrow there’ll be a double feature: terrible parody song lyrics with matching cheesecake!

My secret fetish is…


…laptop backups. I have four independent backups of my current laptop (2x Time Machine, 2x SuperDuper!), one of which is stored on a RAID 6 NAS that backs up to a second RAID 6 volume. Additionally, all source code and blog entries are under source control (Mercurial) and get pushed to a remote server; the blog also gets rsynced to two virtuals in different parts of the country. Which get backed up to the NAS.

I mention this because I just found two external drives containing full backups of my great-grand-previous laptop, plus the actual SSD pulled from it when the board failed. And then I checked the NAS, and found a disk image made from that SSD and a VMware virtual made from it.

For some reason, it doesn’t go over well at work when my first response to someone with a dead computer is, “how many hours ago was your last full backup?”

The good news is that the requirement that we encrypt laptops for people in sensitive positions provided the leverage we needed to get a PO signed for a centralized laptop backup service. Which proved itself pretty darn quickly.

Pity there’s still no cure for people who think that it’s smart to conspicuously put your laptop bag into the trunk of your car when you arrive at a restaurant in the middle of Silicon Valley…

Vacation, all I ever wanted…


“Happiness is a warm server room.”

No, wait, that’s not right.

“Happiness is rooftop building maintenance that interrupts your server-room cooling, with portable chillers that just aren’t cutting it, followed by a surprise UPS failure that takes down all your servers. While you’re out of the country.”

Could be worse. Instead of canceling our visit to a temple flea market and a shrine festival this morning, it could have happened on Friday afternoon…

Craft Sake Week at Roppongi Hills is a brand-new event. We walked over from our hotel in Shiba, got there just before it opened at noon, chatted with the woman running the show (a charming New Zealander who wanted to make sure the limited foreign-language support she could offer was enough for people), and then spent the next four hours drinking glass after glass of really terrific saké.

Thanks to my sister’s well-honed talent for hitting it off with strangers anywhere in the world, we hooked up with a mother and daughter who were on the latter’s final Spring Break vacation before starting her Master’s program in Economics in London. After working our way through the available offerings, the four of us wandered over to the Kit-Kat Pairing Bar, which used “AI” (no actual AI were harmed by this marketing stunt) to pick the right combination of seasonal Kit-Kat and saké for each of us. Ally, the daughter, promptly cheated and ran through the questions again when she didn’t like the result. We approved.

All told, we spent over four hours drinking and chatting, leaving me drunk enough to feel it, and my far-less-massive sister well into wheeeee! territory. The walk back to the hotel involved much greeting of random pedestrians, a bit of stumbling and weaving, and some cat-herding on my part. The day ended early.

Next up, Saturday with penguins, gyoza, a shinkansen ride, and more gyoza (because our neighborhood tonkatsu curry udon joint was closed for the day).

Meanwhile, I watch our group’s Slack channel for news that the UPS is fixed and the folks on-site can bring up enough infrastructure for me to VPN in and do some sadly-necessary work.

Update

Surprisingly clean recovery, although the UPS required a visit from an electrician to get it back online again, delaying things enough that we got to the flea market a bit later in the day, which made for a somewhat sweatier shopping experience (highs up to 78 this week in Kyoto, with humidity to match).

My knees, shins, calves, and right ankle are vigorously expressing their disapproval of all the walking, while my feet are just in a “we’ll get you for this later, dude” mood. All are responding nicely to a felbinac/menthol lotion I discovered on an earlier trip and picked up as soon as we got here.

Foodwise, we’ve struck out twice trying to visit the katsu curry udon place (pro tip: if you’re going to be closed for multiple days, don’t just put 本日 (“today”) on the sign apologizing for it and leave it up for several days).

Finding Tiger Gyoza Hall more than made up for it. The Pukkuri Gyoza in particular were so good that we were tempted to say “mata ashita” on the way out. And, yes, my sister hit it off with two charming older men who spoke decent English and drank heavily, and it turned out one of them had lived in both Chicago and San Jose. Despite being named “thousand winters”, he confessed to preferring Silicon Valley’s weather over Chicago.

(but we’re still going to try for the katsu curry udon again…)

Stop. Just…stop.


I forgot to bitch about this when I first saw it…

New in Emacs 26.1:

** The Emacs server now has socket-launching support. This allows
socket based activation, where an external process like systemd can
invoke the Emacs server process upon a socket connection event and
hand the socket over to Emacs. Emacs uses this socket to service
emacsclient commands. This new functionality can be disabled with the
configure option '--disable-libsystemd'.

** A systemd user unit file is provided. Use it in the standard way:
'systemctl --user enable emacs'.

Honestly, I never saw the attraction of emacsclient in the first place. I open text editors in terminal windows, like Zod intended, and I edit text files in them. My entire .emacs file is devoted to turning off all ‘features’ unrelated to editing text.

Update

I should mention that Emacs also has launchd integration on the Mac, which I’ll never use. It’s the systemd part that bugs me; it’s like what you’d get if you crossed kudzu and cockroaches. Note: do not mention this within earshot of Lennart Poettering. He might try it!

Dear systemd,


ProtectHome considered harmful.

Seriously, WTF? I looked at four recently-kickstarted CentOS 7.x servers and said, “hey, /home ended up on the small partition, so I’ll move it to a bigger one”. I could not do this.

Removing this bullshit from two daemon configurations (NetworkManager? chronyd?) and rebooting managed to fix it on two of them, but not on the other two, and they were all kickstarted with the same config (not a great config, but it wasn’t done by me, and blowing them away and starting over would undo recent work by external con$ultant$).

Dear Gitlab,


After some unknown action on your server has silently deleted most repo/wiki directories for a group (~git/git-data/repositories/$group/$project.git), how do I tell it that I have restored the data from my hourly backups?

Currently it shows “The repository for this project does not exist”.

Honestly, it looks like something tried to delete the entire group and aborted 2/3 of the way through.

Update

Ah, the answer is gitlab-rake cache:clear; now, about how they were deleted in the first place…

Dear Xoratmusoqxee,


Given the recent news about large dumps of user-account data from various hacked sites, I downloaded the full list of records for my mail email domain from HaveIBeenPwned, and found nothing new and interesting. Just the adobe, linkedin, kickstarter, and dropbox hacks from several years ago.

Oddly, none of the email addresses used by Honor Hacker and friends in attempts to extort bitcoin show up in their DB, even though one of those was actually a legit closed account (I briefly had a Livejournal account for commenting, with a unique name and strong password, and the “hacker” included the correct password).

The amusing one was that the “Onliner Spambot” collection from 2017 had a confirmed hit for user “xoratmusoqxee” at my domain. That one doesn’t even show up in my spam, despite being at least as plausible as “hand04”, “quinones12”, “bain66”, “Donnell4Stark”, or the ever-popular “ekgknmfylvtl” (seriously, my spam folder gets daily messages directed to that username, all of them in Japanese).

“P4V considered harmful…”


…to my sanity.

Manager set up a Perforce client on his Windows box, then we changed the directory that was set for its root. We could not get p4v to use the new directory. Even deleting the workspace, restarting the client, refreshing the workspaces, and creating a brand new workspace with the same name didn’t work. It still thought the files should be located in the non-existent directory from the earlier incarnation of the client.

We had to use a different client name to avoid this over-aggressive local cache of data it had no business caching in the first place.

Also, to make the process more funtedious, the client-editing window kept spontaneously resizing itself to be slightly taller than the screen, every time we opened it or tried to resize it to fit.