This is actually a bit disturbing…

The Happy Fan:

The Miller data-manipulation tool has a lot of potential, but the organic development process has left it with a lot of rough edges.

My first hint that all was not well internally was when I was hacking on PDQ output and realized that I had to type nest --implode --values --across-records --nested-fs space to get what I wanted. The manpage includes a shortcut for the nest verb’s other mode, so that --explode --values --across-records --nested-fs can be replaced with --evar, but there’s no matching --ivar. So I forked the project, added it, and sent a pull request.

The functionality was trivial, but the single-line usage description had to be added five times. That sent up a little red flag.

Still, it was easy to do, so I thought I’d see if it was feasible to fix one of the other things that bugs me, which is the lack of quoting and/or escape characters in its native DKVP file format (delimited key-value pairs, aka foo=1,bar=2,baz=3).

The answer turns out to be “not easily”, and I quickly learned some unpleasant things about how it handles other data-format conversions.

Given the perfectly-ordinary CSV input file foo.csv containing:

a,b,c,d,e
"1,1","2
2","3","4\r4","5\n5"

I get the following results:

# DKVP: useless as expected
% mlr --icsv cat foo.csv
a=1,1,b=2
2,c=3,d=4\r4,e=5\n5

# (note: internally, fields have correct values)
% mlr --icsv --ojson cut -f a,b foo.csv
{ "a": "1,1", "b": "2\n2" }

# CSV: reasonable, but strings converted to ints
% mlr --csv cat foo.csv
a,b,c,d,e
"1,1","2
2",3,4\r4,5\n5

# Quoted CSV: better, but should be default
% mlr --csv --quote-original cat foo.csv
a,b,c,d,e
"1,1","2
2","3","4\r4","5\n5"

# JSON: reasonable, but strings converted to ints
% mlr --icsv --ojson  cat foo.csv 
{ "a": "1,1", "b": "2\n2", "c": 3, "d": "4\\r4", "e": "5\\n5" }

# Quoted JSON: oh, hell no
% mlr --icsv --ojson --jvquoteall cat foo.csv 
{ "a": "1,1", "b": "2
2", "c": "3", "d": "4\r4", "e": "5\n5" }

Note that the automated num-ification has real consequences for data processing, since you can’t do things like regex matches or string-substitutions on a number type, and have to explicitly coerce fields back to strings; the error message for this is less than clear. Also, leading zeroes trigger octal conversion…

So that’s an enhancement request for escaping comma, cr, and lf in DKVP, plus a bug on the busted output when you add the --jvquoteall option to avoid the num-ification of string literals. (and it bothers me that I had to explain the bug in a completely different way because one of the devs didn’t understand my sample CSV file…)

I see a massive refactoring in its future (“cover a wall with color-coded sticky notes, then break out the chainsaws and forklifts”). Oh, well, useful tool when used with care.

Cop Craft spoilers, ho.

more...

No idea where all those zombie girls came from; I must have something on my mindbraaaaiiinzzz.

more...

This couldn’t possibly go wrong.

After last week’s Dire Menace From Another Time (with a side order of foreshadowing), it’s time to kick back, relax, and develop some characters! This story was the second half of book 2, and it feels pretty wrapped-up to me, which means they’re really pushing to get through the source material. At least it worked pretty well this time, the relationship between Our Heroes is solidifying, and Tirana got some entertaining culture shocks. Still no sign of the cute elf photojournalist from the OP, who isn’t in any of the light-novel illustrations, so I don’t know when or where she shows up.

Unrelated, I’d like to state for the record that I have never had any information about alleged financial, sexual, or criminal acts undertaken by or on behalf of anyone related to, employed by, or associated with Bill and/or Hillary Clinton. Never never. Absolutely sure of it.

To augment the relatively small amount of Open Source software included with MacOS (soon to be even smaller), I used to use Fink. Then I used to use DarwinPorts (now MacPorts). And just now, I started wondering if it’s time to give up on Homebrew.

Why? Because the maintainers very very quietly decided to gut the functionality of the tools by removing all of the compile-time options for every package, restricting them to a single flavor. The dozens of bugs filed since then just get closed with rude noises, and maybe a handful of the requests for lost functionality get added back as part of the one-and-only build flavor (but only if they’re submitted in precisely the correct format and don’t conflict with other undocumented policies). The best part is that if you had installed something with options selected, it silently broke the next time it was updated.

Their recommended solution: “create your own tap”, which is like telling someone who came in for an oil change to build their own fucking garage if they don’t want 5W30.