There are several lessons to be learned from the Samsung Blu-ray player fiasco, in which pretty much their entire product line turned into a useless pile of e-waste.

1. You don’t know what your Internet-connected appliances are doing, and the manufacturer won’t tell you. Customer service probably doesn’t even know about most of it.

2. The people designing your appliances often don’t think about or thoroughly test boot or update processes.

3. Always mount a scratch monkey.

4. XML makes a terrible config-file format. Ditto YAML and Apple’s Plist format (both of which are just as complex and unforgiving as XML).

When I was at WebTV, every client release meeting included someone who had precise statistics on how many devices were bricked by each previous release, how much it cost to replace them, and the effect on customer churn. This neatly negated the efforts by development and marketing to take shortcuts with QA.

On the service side, we were usually able to just roll back to a previous code or content release within a few minutes of detecting a problem, but there were occasional out-of-band updates, as well as external dependencies. One that bypassed QA one night was an update to the XML config file that controlled ad rotation on the home page. As each ad server retrieved the new file and parsed it, they locked up. When I traced the appropriate process, I saw it spinning in a tight loop trying to parse a comment; someone had manually removed one ad from the rotation. At least, that’s what they thought they’d done, with their limited understanding of XML syntax.

In our case, the code checked for errors, but never got there because it was stuck in an infinite loop; the Samsung startup code simply didn’t check for errors. If the file was syntactically valid, of course it must be semantically valid.

Comments via Isso

Markdown formatting and simple HTML accepted.

Sometimes you have to double-click to enter text in the form (interaction between Isso and Bootstrap?). Tab is more reliable.