August 2006

Forget the mech, send me the pilot...

Of course, to truly pilot this mech, she should be wearing a sailor suit…

The Time Machine paradox

[disclaimer: developers who didn’t attend WWDC don’t have copies of the Leopard beta yet, and when they do send me one, I won’t be able to discuss how things actually work, so this is based solely on what Apple has stated on their web site]

When I heard the initial description of Apple’s upcoming Time Machine feature, it sounded like it was similar to NetApp Filer snapshots, or possibly the Windows volume shadow copy feature that’s been announced for Vista (and is already in use in Server 2003). The answers, respectively, are “not really” and “yes and no”.


The first time you attach an external drive to a Mac running Mac OS X Leopard, Time Machine asks if you’d like to back up to that drive.

Right from the start, Time Machine in Mac OS X Leopard makes a complete backup of all the files on your system.

As you make changes, Time Machine only backs up what changes, all the while maintaining a comprehensive layout of your system. That way, Time Machine minimizes the space required on your backup device.

Time Machine will back up every night at midnight, unless you select a different time from this menu.

With Time Machine, you can restore your whole system from any past backups and peruse the past with ease.

The key thing that they all have in common is the creation of copy-on-write snapshots of the data on a volume, at a set schedule. The key feature of NetApp’s version that isn’t available in the other two is that the backup is transparently stored on the same media as the original data. Volume Shadow Copy and Time Machine both require a separate volume to store the full copy and subsequent snapshots, and it must be at least as large as the original (preferably much larger).

NetApp snapshots and VSC have more versatile scheduling; for instance, NetApps have the concept of hourly, daily, and weekly snapshot pools that are managed separately, and both can create snapshots on demand that are managed manually. TM only supports daily snapshots, and they haven’t shown a management interface (“yet”, of course; this is all early-beta stuff).

VSC in its current incarnation is very enterprise-oriented, and it looks like the UI for gaining access to your backups is “less than user-friendly”. I’ve never seen a GUI method of accessing NetApp snapshots, and the direct method is not something I’d like to explain to a typical Windows or Mac user. TM, by contrast, is all about the UI, and may actually succeed in getting the point across about what it does. At the very least, when the family tech-support person gets a call about restoring a deleted file, there’s a chance that he can explain it over the phone.

One thing VSC is designed to do that TM might also do is allow valid backups of databases that never close their files. Apple is providing a TM API, but that may just be for presenting the data in context, not for directly hooking into the system to ensure correct backups.

What does this mean for Mac users? Buy a backup disk that’s much larger than your boot disk, and either explicitly exclude scratch areas from being backed up, or store them on Yet Another External Drive. What does it mean for laptop users? Dunno; besides the obvious need to plug it into something to make backups, they haven’t mentioned an “on-demand” snapshot mechanism, simply the ability to change the time of day when the backup runs. Will you be able to say “whenever I plug in drive X” or “whenever I connect to the corporate network”? I hope so. What does it mean for people who have more than one volume to back up? Not a clue.

Now for the fun. Brian complained that Time Machine is missing something, namely the ability to go into the future, and retrieve copies of files you haven’t made yet. Well, the UI might not make it explicit, but you will be able to do something just as cool: create alternate timelines.

Let’s say that on day X, I created a file named Grandfather, on X+20 a file named Father, and on X+40 a file named Me. On X+55, I delete Grandfather and Father. On X+65, I find myself missing Grandfather and bring him back. On X+70, I find myself longing for a simpler time before Father, and restore my entire system to the state it was in on X+19. There is no Father, there is no Me, only Grandfather. On X+80, I find myself missing Me, and reach back to X+69 to retrieve it.

We’re now living in Grandfather’s time (X+29, effectively) with no trace of Father anywhere on the system. Just Me.

Now for the terror: what happens if you set your clock back?

First they hit, then they run

Driving home from class tonight, someone hit my car. His lane was closed ahead due to construction, and I guess he decided that he’d rather merge in front of me than behind me. Sadly, he ended up trying to merge through me at about 75-80 MPH, and then decided to run for it. I caught up to him long enough to get his license plate number, and then he was gone (making a fast exit from 101 South to 85 South, then running a red light to turn onto Central Expressway South).

Worse news for him: a witness pulled over and stayed to make a statement to the police, and his memory of the incident was a good match for mine.

The damage? Several deep, long scratches along the driver’s side, from about the side mirror forward to the front of the wheel well, with the finish scraped off of the tire rim in several places. The witness said my car was pushed to the right about a foot and a half by the impact, and he was surprised that I wasn’t hurt. No apparent mechanical damage, and it drove home fine.

I’m fine, and I have excellent insurance, so even if the sorry bastard is uninsured, all will be well. It could have been a lot worse. In the cop’s experience, an impact like that at that speed could have easily caused my front tires to lose traction with the road, sending my car rolling sideways down the highway. He figured that the relative size and mass (my “small” SUV versus his smallish hatchback/whatever) are what saved me.

The (surprisingly small) damage. Aside from the scratches, the door rubs a bit when you open and close it:

car damage

[update 10/11/2006: My insurance company ran the license plate number and came up with a 2000 four-door Volkswagen in Sunnyvale, color unknown. Given the location of the accident, that suggests that I got the number right.]

は vs. が

The most coherent explanation I’ve found for understanding the difference between the “wa” and “ga” particles in Japanese comes from Jay Rubin’s book Making Sense of Japanese (originally titled “Gone Fishin’”). Namely, use “ga” when you want to emphasize the previous word, and “wa” when you want to emphasize the following words. Which to use depends on what kind of question you’re asking or answering.

For example, he gives the following answers:

  1. Ikimashita. "I went."
  2. Watashi wa ikimashita. "Me? I went."
  3. Watashi ga ikimashita. "I went."

And these matching questions:

  1. Dou shimashita ka. "What did you do?" Or: Ikimashita ka. "Did you go?"
  2. Soshite, Yamamura-san wa? Dou shimashita ka. "And now you, Mr. Yamamura. What did you do?"
  3. Dare ga ikimashita ka. "Who went?"

And that’s why 「私はケーキです。」 does not mean “I am the cake”, no matter what BabelFish says. Except when it does, of course, but for that, you’ll have to read the rest of the book.

"...where were you going in such a hurry?"

17 Egyptian exchange students, all headed to Bozeman, Montana. Six show up as scheduled, the rest are eventually located in: Richmond, Virginia; Minneapolis, Minnesota; Manville, New Jersey; Dundalk, Maryland; O’Hare International Airport; and Des Moines, Iowa.

Money quote:

"None of the students is considered a terrorism risk."

I’d feel a little more confident about this statement if they’d been found in, say, a Las Vegas casino hotel in a room filled with booze and strippers.

So where’s Dundalk, anyway? Why, it’s the home of the Dundalk Marine Terminal, whose major clients include the National Shipping Company of Saudi Arabia. Lots of bulk cargo coming in through there, from all over the world.

Manville just seems to be a wide spot in the road that’s half an hour away from the Newark Liberty International Airport and McGuire Air Force Base, the sort of neighborhood where one death every two-three years changes the murder rate from 0.0 to 9.7.

I’d like to think that there’s an innocent explanation for all this. I just can’t think of one.

They still don't get it...

[last update: the root cause of the Linux loopback device problem described below turns out to be simple: there’s no locking in the code that selects a free loop device. So it doesn’t matter whether you use mount or losetup, and it doesn’t matter how many loop devices you configure; if you try to allocate two at once, one of them will likely fail.]

Panel discussions at LinuxWorld (emphasis mine):

"We need to make compromises to do full multimedia capabilities like running on iPod so that non-technical users don’t dismiss us out of hand."

"We need to pay a lot more attention to the emerging markets; there’s an awful lot happening there."

But to truly popularize Linux, proponents will have to help push word of the operating system to users, panelists said.

... at least one proponent felt the Linux desktop movement needed more evangelism.

Jon “Maddog” Hall, executive director of Linux International, said each LinuxWorld attendee should make it a point to get at least two Windows users to the conference next year...

I’m sorry, but this is all bullshit. These guys are popping stiffies over an alleged opportunity to unseat Windows because of the delays in Vista, and not one of them seems to be interested in sitting down and making Linux work.

Not work if you have a friend help you install it, not work until the next release, not work with three applications and six games, not work because you can fix it yourself, not work if you can find the driver you need and it’s mostly stable, not work if you download the optional packages that allow it to play MP3s and DVDs, and definitely not work if you don’t need documentation. Just work.

[disclaimer: I get paid to run a farm of servers running a mix of RedHat 7.3 and Fedora Core 2/4/5. The machine hosting this blog runs on OpenBSD, but I’m toying with the idea of installing a minimal Ubuntu and a copy of VMware Server to virtualize the different domains I host. The only reason the base OS will be Linux is because that’s what VMware runs on. But that’s servers; my desktop is a Mac.]

Despite all the ways that Windows sucks, it works. Despite all the ways that Linux has improved over the years, and despite the very real ways that it’s better than Windows, it often doesn’t. Because, at the end of the day, somebody gets paid to make Windows work. Paid to write documentation. Paid to fill a room with random crappy hardware and spend thousands of hours installing, upgrading, using, breaking, and repairing Windows installations.

Open Source is the land of low-hanging fruit. Thousands of people are eager to do the easy stuff, for free or for fun. Very few are willing to write real documentation. Very few are willing to sit in a room and follow someone else’s documentation step-by-step, again and again, making sure that it’s clear, correct, and complete. Very few are interested in, or good at, ongoing maintenance. Or debugging thorny problems.

For instance, did you know that loopback mounts aren’t reliable? We have an automated process that creates EXT2 file system images, loopback-mounts them, fills them with data, and unmounts them. This happens approximately 24 times per day on each of 20 build machines, five days a week, every week. About twice a month it fails, with the following error: “ioctl: LOOP_SET_FD: Device or resource busy”.

Want to know why? Because mount -o loop is an unsupported method of setting up loop devices. It’s the only one you’ll ever see anyone use in their documentation, books, and shell scripts, but it doesn’t actually work. You’re supposed to do this:

LOOP=`losetup -f`
losetup $LOOP myimage
mount -t ext2 $LOOP /mnt
umount /mnt
losetup -d $LOOP

If you’re foolish enough to follow the documentation, eventually you’ll simply run out of free loop devices, no matter how many you have. When that happens, the mount point you tried to use will never work again with a loopback mount; you have to delete the directory and recreate it. Or reboot. Or sacrifice a chicken to the kernel gods.

Why support the mount interface if it isn’t reliable? Why not get rid of it, fix it, or at least document the problems somewhere other than, well, here?

[update: the root of our problem with letting the Linux mount command auto-allocate loopback devices may be that the umount command isn’t reliably freeing them without the -d option; it usually does so, but may be failing under load. I can’t test that right now, with everything covered in bubble-wrap in another state, but it’s worth a shot.]

[update: no, the -d option has nothing to do with it; I knocked together a quick test script, ran it in parallel N-1 times (where N was the total number of available loop devices), and about one run in three, I got the dreaded “ioctl: LOOP_SET_FD: Device or resource busy” error on the mount, even if losetup -a showed plenty of free loop devices.]

fun with tongues


Worst. Nutritional. Advice. Ever.

Pardon me while I point and laugh:

High water intake reduces fat deposits and rids the body of toxins. Simply drinking eight 16 oz. glasses of water throughout the day, cooled to 40 Fahrenheit, will burn 200 calories; that's equivalent to running 3 miles!

I have no words.

Random cuteness

[update: to no surprise, the rights-holders in Japan have finally caught up with Youtube, and forced the removal thousands of video clips. I’m not upset with them about it, particularly for things available on DVD (I own import copies of all of the concert and PV footage I linked below); I just wish it were possible to legitimately watch the ephemera.]

This is just a placeholder for links to random videos on Youtube:


How not to move servers

Tip for the day: when you’ve arranged for a professional computer moving company to relocate 30 critical servers from one state to another, and the driver shows up alone in a bare panel truck, without even a blanket to keep the machines from bouncing around on their way to the warehouse, do not let him take them.

The driver was as surprised as I was, perhaps more so. He thought we had a shrink-wrapped pallet of boxes that could be popped onto the truck and dropped off after he made a few more stops. The dispatcher tried to talk him into loading the stuff loose. The dispatcher tried to talk me into letting the driver load the stuff loose, swearing that it would be fine for the short trip to the warehouse.

Things went downhill from there.


Current stats across all my email accounts: 115 spams/day, 30 of them caught by Mac Os X’s Junk filtering, 85 ended up in my Quarantine folder (people who aren’t in my address book or email history).

Best news: 0 false positives in the most recent ten days worth of email. By its nature, the Quarantine folder will occasionally contain real email from someone who’s trying to reach me for the first time, but I have a variety of other rules that do a pretty good job of detecting common cases. I also create custom addresses for every new vendor and site membership, and a monthly disposable address for this blog, and all of those get filtered into special folders.

I haven’t set up anything more sophisticated, largely because the problem is under control. About twice a month, something makes it into my real Inbox, and that’s not worth the effort.

Pedestrian On Pavement

I got a ticket yesterday. More precisely, I got a fake ticket yesterday, because it was the only way for the cops to get the crazy angry person to shut up and go away.

I had a little work project that was kind of important. Namely, I needed to get over half a million dollars worth of servers packed up and loaded onto a truck (the same ones that were supposed to be shipped out on Friday). To do that, we needed to park the truck. Unfortunately, just as we were pulling into the commercial loading zone that we’d been patiently waiting for for twenty minutes, some clown in an SUV whips around the truck and starts backing into it.

I stepped out into the street and waved him off. He kept coming, until his bumper was about three inches from my body. Then he jumped out and furiously accused me of trying to steal his parking space, shouted at everyone within reach (including a completely unrelated moving company that was working across the street), and then ran off claiming he was going to find a cop to take care of me, leaving his car blocking both the parking spot and the street.

We found a cop first. When he returned with his dry-cleaning (he later claimed he really was making a commercial delivery, but that box never left the back of his SUV, and the cop saw him picking up the suit…), she was already writing up his ticket, and informing him that he was two minutes away from being towed.

He shouted at her. He shouted at us. He shouted at her sergeant, when he showed up. He harangued the bums on the sidewalk, telling them what horrible people and criminals we were. He tried to get the cop to give my truck driver a ticket for blocking the road. He tried to get the cop to give me a ticket for illegally attempting to reserve a parking space.

He got several tickets, which he’ll have to pay for. To shut him up, they wrote out a phony ticket for me, which will be dismissed when the cop deliberately fails to appear in court (her exact words: “this is bullshit, don’t pay it”). He tried to get my name so he could go after me personally, and the cop patiently explained that he had no right to that information.

And to think that this was actually better than my day Friday, which involved the world’s most carelessly ambitious contract Unix sysadmin trying to get me to let him work unsupervised as root on a production server that I’m responsible for (“Hi, Mark!”).

Stupid Apple Tricks

YouTube stopped working a while back on my Quad G5. Videos played, but with no sound. All other forms of video worked fine, and the exact same video would work if I downloaded it manually and ran it from a different FLV player.

I reinstalled Flash, I wiped caches to make sure I didn’t have an old version of the YouTube flash player, grabbed the latest version of Firefox and tried it there, etc, etc. No change. It wasn’t a big problem, since I usually don’t surf from that machine, but I finally spent a few minutes chanting Google incantations and found the answer:

Run the Audio MIDI Setup utility and change the Audio Output Format back to 44100 Hz

This just in: Ubuntu is crap

Well, at least in the area of configuration, maintenance, and release management, the current version shows its dark roots. Before anyone speaks up, I’ll say that I’m generally happy with using FC5 and RedHat Enterprise on our servers at work, but someone had recommended Ubuntu server as a possible base OS for virtualizing my personal machine with VMWare Server.

It installed correctly, but wouldn’t boot. The solution I located required the following steps:

  1. boot from the install CD
  2. discover the command-line option for booting in rescue mode
  3. guess the partition name to mount as / for rescue
  4. open a root shell
  5. manually add a nameserver to /etc/resolv.conf
  6. type apt-get install linux-686
  7. say yes to all the dependencies this requires
  8. watch and wait
  9. reboot

“Fixed in next release,” supposedly, but between that early-warning sign and some of the obvious eccentricities I tripped over, I don’t think I’ll bother with it.

“Need a clue, take a clue,
 got a clue, leave a clue”