More fun than one man can handle...


Sometime this morning, someone rebooted a KVM server. We don’t know who, yet, but this would only have been a minor problem if it weren’t for the fact that another unknown someone had accidentally deleted the disk images for some of the VMs running on that server.

A month ago. No one noticed because they kept running on the unlinked open files…

Daily full backups are your friend!

Update

NetworkManager: threat or menace?

Seriously, who even configures a server with it, and who came up with the idea of instantly taking down the interfaces the moment you save ifcfg-eth0 to switch from NM to static config? Fortunately, IPMI meant that I didn’t have to physically plug a monitor into the server to get back in.

(although Neal did have to plug in the IPMI interface for me; the perils of setting up new servers in a hurry…)


Comments via Isso

Markdown formatting and simple HTML accepted.

Sometimes you have to double-click to enter text in the form (interaction between Isso and Bootstrap?). Tab is more reliable.