I hate it when fixing one problem breaks something else, especially when it’s subtle.

A few weeks ago while testing our new IPSec VPN connections to external partners, we discovered that I could ssh/scp through the VPN from my Macs, but none of our Linux boxes could, and another Mac running allegedly-identical software had horrible performance issues.

The fix was a change in the OpenBSD firewall that also served as the IPSec endpoint: scrub reassemble tcp. The problem went away like magic.

Today, we found out that there’s a single external partner we have to post some data to via an HTTPS connection, and it worked fine from machines outside of our firewall, but failed about 50% of the time from all the machines inside our firewall.

…except for my Macs, which worked 100% of the time. I fired up a CentOS 5 Parallels session on one of them, and it failed 50% of the time. Surely it couldn’t be…

It was. Remove the scrub line, and the HTTPS post worked from everywhere, but now my IPSec VPNs were hosed again.

So:

scrub from any to $IPSEC1_INT reassemble tcp
scrub from any to $IPSEC2_INT reassemble tcp
scrub in

The root cause appears to be the partner’s IIS server failing to properly implement RFC 1323, causing some of the fragmented packets to be rejected during reassembly.

When Adobe released the CS suite, they added a revision control system called Version Cue. I had mixed feelings about it, but at least it was off by default.

When they released the CS2 suite, they turned it on by default, without any regard for security. I was less than thrilled:

The only nice thing I can say about it is that it doesn’t add a new rule to the built-in Mac OS X firewall to open up the ports it uses.

Care to guess what CS3 does? If you guessed “adds a new firewall rule”, you’d only be half right. It adds a new firewall rule, and then turns off the firewall. That part’s a mistake, obviously, but silently modifying your firewall settings to turn on an unsecured file server is unforgivable.

[Update: Adobe acknowledges their mistake in turning off the firewall, but does not apologize for silently turning your machine into a server and sharing your documents]

This is a remarkably useful gadget that’s paid for itself several times in the past month. What it does: connect an IDE or SATA drive via USB2 without putting it in an enclosure. It’s faster to work with the bare drive when you just need to grab some data from a failed machine or scrub a disk before reuse or service.

Mossberg is right about how much crapware infects a brand-new Sony laptop running Vista, and that’s sad. Because Vista sucks a lot less than Windows XP, and it deserves better.

Despite all of the stuff that got cut from the release, Vista isn’t just another minor update to the ancient NT codebase. There are serious architectural changes that make it an honest-to-gosh 21st Century operating system that will produce a better user experience on newer hardware, once every vendor updates their software to use the new APIs.

Microsoft has done a lot of good, solid work to improve not just the use, but also the installation. Rory and I both did fresh installs of Vista Ultimate (onto MacBooks…), and felt that the install process was on par with Mac OS X, if not a little better in places. I’m still installing XP at least once a week, and I can’t tell you just how significant an improvement this is. [don’t ask about Linux installs, please; I just ate]

Is the Aero UI gaudy and gratuitous? Yes. Are the menus and control panels different from previous Windows in ways that aren’t obviously functional? Yes. Does any of that really matter after about twenty minutes of familiarization? No, not really. I expect the adjustment period to be pretty short for most users, and none of them will ever want to use an XP machine again, just like most Mac users were delighted to abandon the limits of Mac OS 9 once they settled into Mac OS X 10.0.

Because that’s what Vista is: Microsoft’s OS X 10.0, with all that that implies. The XP compatibility is a subtler version of Apple’s Classic environment, and they really, really want everyone to rewrite their software to use the modern APIs that, for instance, use “fonts” instead of “bitmaps”. There’s going to be a few years of mostly-compatible legacy apps, service packs that break random things as a side effect of improving performance and reliability, and general chaos and confusion. And because it’s Microsoft, they’re going to try to solve the problems faster by throwing more engineers at them, which never works out well.

In the end, though, Vista will have 90% of the desktop market, Mac OS X will have 9.99% of it, and the rest will be evenly divided between fourteen different Linux distributions that don’t ship with all of the drivers you need, but they’re free and you control everything and you can fix it yourself and it even has Ogg Vorbis support.

Office 2007, on the other hand, is a major upgrade hassle, and it has nothing to do with functionality or cost. Microsoft’s grand release plan failed to cope with one very significant fact: experienced Office users know where everything is, and spend far more time navigating Office menus than they do Windows menus.

We’ve been forced to start slowly rolling it out at work, and it’s painful. Everyone who gets it hates it, because they need to get their work done right now, and they don’t have time to go to a retraining class and learn the joys of the ribbon and the “obviously superior” new arrangement of commands and menus. They don’t care about Vista; they just find the Start button, select “Word” or “Excel”, and they’re happy. But when the Word and Excel interfaces change in fundamental ways (and, worse, ignore the settings that are supposed to make sure files are saved in Office 2003 formats…), they’re angry and frustrated.

[Side note to Gerry: read the preceding paragraph carefully three times, and then shut the fuck up about OpenOffice.org as an alternative. It’s not better, it’s not a complete suite, it’s not as compatible, it adds to my support load, it requires just as much retraining effort, and I can’t hand the users Dummies books and send them off to training classes, which we don’t have time for anyway because we’re a startup in the middle of a major product launch. Got it?]

If we’d had the money a few months ago, we could have picked up the volume license agreements that would let us avoid Office 2007 for another year. And we’d have been a lot happier, because “launching your first product” is not the time to cut into everyone’s productivity by changing their tools.

Rory has ranted a bit about our recent laptop troubles. After giving up on those two companies, and not being able to fit ThinkPads into the budget, we looked for an alternative. These days, we’re also constrained by the desire to avoid becoming a mixed XP/Vista shop, so I went to the vendor who likes us the most, PC Connection, and sorted through their offerings.

The first “fix me now” user really, really wanted a lightweight machine, and had a strong affection for Bluetooth, so we bought him a Sony VAIO SZ340P and bumped the memory to 1.5GB. He loves it, and I was pretty pleased with the out-of-the-box experience as well (including their new packaging). There are only three real problems: it takes half an hour and three reboots to delete all of the crapware that’s preinstalled, you have to spend an hour burning recovery DVDs because they don’t ship media, and the default screensaver plays obnoxious music on a short loop.

The second user liked the SZ340P, but wanted something even lighter, so we bought her the SZ360P. It’s a quarter-pound lighter, uses the same docking station (which ships without its own power supply, but uses the same one as the laptop), and is also a really nice machine.

The downside of 4-pound laptops is they’re not as sturdy, so for the next four new-hires in line, I looked for something a little bigger, and ended up choosing the BX640P, with RAM bumped to 2GB. Different docking station (nicer, actually, with room for an optical drive and a spare battery to keep charged), different set of crapware, and not a widescreen display, but a better keyboard and a sturdier feel, and I’m equally pleased with its performance.

The only serious negative: it looks like the BX series will be discontinued, so when they run out and I need to start buying Vista machines, I’ll have to switch series. At the moment, I’m leaning a little toward the FE890 series, but PC Connection doesn’t stock the full range yet, so I can’t get the CPU/RAM/disk combination I want. With luck I can put that off for a few months, though.

With the previous brand, 2 of five had video and wireless problems. The five VAIOs I’ve set up so far have been rock-solid, and I expect the same from the other three that just arrived.

Sadly, while we’ll be able to put off the Vista migration for a little while (hopefully until Juniper gets their VPN client working…), Microsoft Office 2003 is a dead product, and starting Monday we’ll have users running 2007. On each user’s machine, I have to open up each Office application as that user, click the unobtrusive button that looks like a window decoration, click on the “{Word,Excel,PowerPoint} Options” button, select the Save tab, and set the “Save files in this format” option to use the Office 97-2003 format. Or else.

[Update: Actually, if you like ThinkPads and you’re willing to buy them right now, PC Connection has some nice clearance deals. If we were a bigger company, I might find the “buy 15, get one free” deal attractive…]

[4/17/2007 update: okay, one of the Sony BX laptops just lost its motherboard, after locking up at random intervals over a week or two. That still leaves 9/10 good ones, which is better than we got with Dell and Alienware.]

[Peem] whispers: wiped in TIB, can u tank 4 us

[Ferendo] whispers: Maybe, what's up?

[Peem] whispers: need 2 clear 4 1st boss, u tank 6 whelps we dps

[Ferendo] whispers: That sounds easy. Go ahead and summon me.

Ferendo joins the party.

[Ferendo] says: Okay, where are the whelps?

[Peem] says: dead ahead, dood

[Ferendo] says: What, past that nest of elite dragonkin?

Peem points at Enraged Harbinger Whelp.

[Ferendo] says: Ah, right in the middle of the nest of elite dragonkin. That’s a problem.

[Peem] says: u said easy

[Ferendo] says: I said tanking six whelps would be easy. Nobody told me about the 18 elite dragonkin fireballing me and healing each other in the middle of the fight. All my fire resist gear put together won’t keep me alive for fifteen seconds in that, and there’s no way you can DPS them down before I die. And when I die, you die.

[Ferendo] says: Look, send a tell to my friend Akamai; he’s got a full set of Molten Core gear and some fire pots, and he could clean this room out with his eyes closed.

[Peem] says: tried, he said no pugs

[Ferendo] says: What’s your repair budget?

[Peem] says: ??

[Ferendo] says: Without the right gear, we’re going to wipe half a dozen times before they’re all down, and that’s going to cost me at least six gold.

[Peem] says: no cash, just got [Gaudy Shiv of the Poser] at AH

[Ferendo] says: Then you’re fucked. Sorry, guys, I’m out of here.

Ferendo leaves the party.

[Peem] whispers: u suck

Ferendo is now ignoring Peem.

I do not like IPSec. I do not understand IPSec. Sadly, cheap VPN routers purchased by external partners to whom we must give some access pretty much speak nothing else. [don’t get me started on packaged SSL VPN servers…]

Fortunately, our firewall runs a recent release of OpenBSD. Even more fortunately, there’s an excellent site on configuring OpenBSD as an IPSec server, including sample PF firewall rules.

I used a recent build of Parallels to set up a private, non-routed network with three virtual servers on it, put one of them on the real network as well, set it up as a firewall and router, and tinkered with a pair of Netgear VPN routers until they both could connect to one of the private servers without seeing the other.

Then I worked on the PF ruleset until I knew I could cut off either Netgear without affecting anything else, and transferred my configuration to our real-world firewall. Works like a charm.

It appears that the best way to use IPSec is to completely ignore all of its management features, set up a generic tunnel config, and handle all the access controls in your firewall. One less convoluted config-file syntax to learn, one less place to screw up and allow the wrong people to get at the wrong stuff.

I think this is the single finest example of premature optimization in existence today. From “Life with djbdns”:

The format of this datafile is documented at http://cr.yp.to/djbdns/tinydns-data.html. It looks a bit strange at first because it is not optimized to be readable by humans, but rather is optimized for parsing.

Following the link turns up this quote, which had me on the floor:

The data format is very easy for programs to edit, and reasonably easy for humans to edit, unlike the traditional zone-file format.

Yes, I think we can all agree that a colon-separated data file where whitespace is illegal and record type is indicated by a single ASCII character (+%.&=@-’^CZ:) is easy for a simple-minded program to edit. I don’t think anyone with two brain cells to rub together can agree that it’s “reasonably easy for humans to edit”.

I must confess that, between his famous Usenet debut and my first look at the daemontools package he inflicts on all users of his software, I have never been particularly open-minded as to the merits of “the DJB way”. I’ve never heard a compelling technical reason for a site to abandon Bind and Postfix, and his advocates tend to have the glassy-eyed stare of veteran kool-aid drinkers, so until recently I hadn’t even bothered to look at his data formats.

The djbdns data file format? Fucking stupid.