2010-05-22T11:59:17+00:00

“Less thinking. More testing.”

May 22, 2010April 21, 2026/stevereads/Leave a comment

(Attention conservation notice: nearly 2,000 words that start with test-driven development in software, skip along to application prototyping, then take a big leap to an attack on libertarianism.)

I’ve meant for a long while to write about Kent Beck’s Test-Driven Development By Example, but — as you can see from this blog in general — I’ve had a lot less time to write recently. The book hasn’t yet changed my life, but it should, and it will. And I think the idea has far broader applicability than just software development, which I’ll try to get into below.

The basic premise of test-driven development is to write your tests before you write your code. The structure is like so:

Write a test asserting something about the code that you’ve not yet written. For instance, if you intend to write code computing the number of days between two dates, you might make a few assertions: that the number of days between a date and itself should be zero; that the number of days between March 1 of 2009 and March 1 of 2012 is one day more than three times 365; and so forth. The more assertions you can make about this as-yet-unwritten code, the better.
Since you’ve not written the code, the assertions will fail. In fact, the code won’t even compile.
Write the simplest version of the code that will pass the test. Write this as quickly as possible.
Tests pass! Joy!
Refactor.
Having accomplished the task you were on, continue to bigger and better things. Go to step 1.
Repeat 1)-6) until you’ve achieved whatever you were trying to do.

The advantages of having unit tests are well known. The virtues of unit tests can best be understood if you know what their absence is like. If you’re like me, you’ve worked before on code bases that had absolutely no tests, and the experience is terrifying. You can’t change one bit of code without worrying that you’ve broken something in some far-off part of the code. If you’re like me, this experience makes work actually unbearable: the more code you dip your fingers into, the wider the potential swath of destruction. Again if you’re like me, this can turn your stomach into a big ulcer which actually makes it hard to sleep. On the wrong day, it can lead you to excessive caution, which keeps you from doing work. Which is bad and makes your bosses hate you. You want your bosses to love you, don’t you? Of course you do.

Imagine instead that the code is entirely covered by unit tests. (This nirvana is known as “100% code coverage.”) Now, if you change the code, you just run the tests. Do all the tests pass? Joy and rapture! You can keep changing code to your heart’s content. When you break a test, figure out why you broke it, fix it, confirm that the tests now all pass, and move on. Continue to add tests for all the code that you add. Again, if you’re like me, this gives you a feeling of calm and confidence, which makes you work faster, which makes your bosses like you more.

Of course, sometimes your code will break for reasons that you didn’t test against. This is unfortunate but expected. When this happens, add another unit test to guard against the heretofore-unanticipated case. In this way, the unit tests document your knowledge about the particular problem domain. If done well, people should be able to understand your code by reading the unit tests. A unit test can essentially be read as “the code is expected to respond like so when it encounters a world shaped like so.”

Striving for 100% code coverage leads you to write smaller functions, because it’s easier to write unit tests to cover a smaller, more-specialized function. This is a happy side-effect: smaller, more-specialized functions are a good thing, whether or not you’re writing unit tests.

Another way that test-driven development contributes to a fearless coding experience is that — per the title of this post — it encourages you to think less and code more. If you’re like me, you can get stuck inside your own mind, wondering whether the particular path you’re going down will work. The TDD approach is to move past this state of mind as fast as you can, by writing tests. Don’t speculate idly about whether your code will do what you expect; think about how it should respond to known inputs, then write code that responds appropriately to those inputs.

Any number of conclusions might come out of this testing discipline:

your speculation proved correct; the code works.
it proved incorrect, and you need to pursue another line of development.
it proved partly correct, partly incorrect, and you need to course-correct.

This institutionalized course-correction is, I think, the greatest virtue of test-driven development. And it’s why some variant of test-driven development applies in much broader contexts.

Take one context that’s only slightly broader, namely the process of building an entire app from the ground up. We recently did this at work; the task for one of our sprints was to build a prototype of an app. I wasn’t entirely prepared for what “prototype” means, but now I think I get it. A few important aspects of prototyping stand out for me from this experience:

Build something with a terrible user interface, but the broad rough structure of what we think users will want. Explicitly do not make it pretty. If you make it pretty, the users who beta-test it and the designers who make it pretty will focus on the visual details rather than on how it functions. In order to keep their eyes on the prize, write a computer program that is only one or two steps up from a sketch on paper.
Write a prototype that exercises the necessary backend code, like databases and API calls and so forth. You might find that your API calls take too long to return, and thereby couldn’t fit into the application that you’re building. Or you might find that your database doesn’t have indexes where it needs them. Or you might find that you need to restructure the entire app to work with APIs and backend databases that are beyond your control.
By putting the code in front of users, you might find that they don’t actually want the program that you envisioned. Or they want it, but they’d really want it if you just added a little something extra.

When I first mentioned “less thinking, more testing” to my friends, one friend raised the absolutely valid point that this approach doesn’t rid you of the need for design. That’s absolutely true. First of all, you need actual hypotheses to put in front of users; you can’t put a blank piece of paper in front of them and ask them to draw what they want. You need to focus their attention in a particular direction. When you’re building the backend architecture, you likewise aren’t starting from a blank slate.

But the point is course-correction as quickly and as often as possible. That fundamental message is why I think test-driven development and rough prototyping are applicable far beyond software development. It’s more than a little applicable to ideologies. Take, for instance, the recent kerfuffle over Rand Paul’s opposition to the Civil Rights Act. Matt Yglesias pulls on this thread and attacks the very idea that adhering to consistent principles even when they drive you off a cliff is somehow admirable. I completely agree. Consistency is a fine virtue, and a belief system that’s not consistent can’t be entirely true. But there are many virtues apart from consistency; among the greatest is non-insanity.

To keep your beliefs from veering off into the insane, you need to course-correct as often as possible. We’re not playing some game where the purpose is to start with reasonable-seeming principles and derive hilarious conclusions that clearly make no sense; the point is to build ethics that work for you in conducting your daily life, and to build policies that work for your countrymen. If it looks like you’ve built a chain of reasoning that led from obvious-seeming premises to ridiculous conclusions, you probably need to reconsider the premises. If Goldwater believed that opposition to civil-rights legislation was obviously true, then the principles were so much chin music to defend conclusions that he would have come to anyway; if, on the other hand, he adhered to those conclusions with great regret because he believed that the principles were correct, then he should take stock of how he arrives at his conclusions.

It’s not exactly new wisdom that blind adherence to principles can lead you astray. Holmes said as much in The Common Law, whose opening words are

The object of this book is to present a general view of the Common Law. To accomplish the task, other tools are needed besides logic. It is something to show that the consistency of a system requires a particular result, but it is not all. The life of the law has not been logic: it has been experience. … The law embodies the story of a nation’s development through many centuries, and it cannot be dealt with as if it contained only the axioms and corollaries of a book of mathematics.

Earlier in my life, I thought it was very important which ethical principles one had. I thought consistency was the most important thing. (Blame college; maybe blame academia more generally? College is a great place to pick up ideas at an impressionable age and run with them until everyone smirks at you with the amusement of the non-cloistered.) I still think it’s important, but there are many more important things. Constantly clinging to reality is among the most important. Constant course-correction, with input from the real world at every branch, is extremely helpful at keeping you moored in reality.

This, by the way, is why I’ve never been able to get far into Nozick’s Anarchy, State, and Utopia. It always feels like a shell game: “Let’s suppose you believe some premises about liberty and side constraints … We’ll just shuffle around the shells a little bit and … here we go: clearly you must believe this thing about government non-intervention.” I no longer trust long chains of reasoning from seemingly self-evident low-level principles. I want principles nowadays that are closer to daily life, whence the jump from them to concrete action is smaller and less fraught with the potential for insanity.

That said, of course there’s room to move in the other direction: I tell you that I believe X and Y, and you reply that X and Y are only instances of Z. (With some fear of stretching an analogy too far, this is akin to refactoring.) So now I believe Z instead, which is a generalized version of X and Y. Or maybe you ask me whether I believe A; if I say yes, you point out that A contradicts X. If I agree with you that there’s a contradiction, I now have a choice: continue to believe X, or change my belief in X. I believe the endpoint of this process is what Rawls called reflective equilibrium.

And of course when you course-correct frequently, you still need principles. Principles help determine the path that you start down, and help determine which experiments to perform to correct your course. But the goal should be to experiment at every possible step.

Am I just calling for more use of the scientific method here? I think I am. It works at small scales like software, and it also works at large scales like philosophy.

svn commit “unable to lock” error under Mac OS X

May 3, 2010/stevereads/2 Comments

This will be interesting to approximately none of you, but I feel I need to spread the knowledge of hell to a wider world.

I will perhaps go into greater detail at a future date about all the *other* problems preceding this one while I used svn (and Maven, and Eclipse) under OS X. They all essentially stem from the fact that OS X’s filesystem is case-insensitive. … Or rather, not case-insensitive: you can’t do

(16:41) slaniel@Steve-Laniels-MacBook-Pro:~$ mkdir test_dir
(16:42) slaniel@Steve-Laniels-MacBook-Pro:~$ mkdir test_Dir
mkdir: test_Dir: File exists

But you can do

(16:42) slaniel@Steve-Laniels-MacBook-Pro:~$ mv test_dir test_Dir
(16:42) slaniel@Steve-Laniels-MacBook-Pro:~$ rm -rf test_dir

Since it’s not entirely case-insensitive, it is *sometimes* — but not always — a problem to have both ‘foo’ and ‘Foo’ in the same svn checkout. More to the point here: what if you have ‘Foo’ and want to rename it to ‘foo’? You’d want to do `svn mv Foo foo`. There will be bequeathed unto you a sadness:

(16:45) slaniel@Steve-Laniels-MacBook-Pro:~/svn/sandbox/slaniel$ svn mkdir foo
A foo
(16:45) slaniel@Steve-Laniels-MacBook-Pro:~/svn/sandbox/slaniel$ svn mv foo Foo
svn: Unable to lock ‘Foo’

The reason it can’t lock ‘Foo’ is that it already has ‘foo’ locked, and it thinks that ‘foo’ is the same as ‘Foo’. So it can’t move, in other words, because the filesystem is case-insensitive.

To redress this just now, I had to do something like

svn mv foo bar &&
svn ci -m “Temporarily moving foo to bar” &&
svn mv bar Foo &&
svn ci -m “Moving bar to Foo”

You need to do the commit after each move; you can’t just do

svn mv foo bar && svn mv bar Foo

Maybe all of this was obvious to all OS X svn users other than me. I assure you that it was *not* obvious to me.

Mac filesystem case-insensitivity just wasted an inordinate quantity of time and money from some of my company’s smartest engineers. I am displeased. Perhaps this post will save someone else some time in the future.

At a later date, after I’ve actually completed some work, perhaps I will explain all the *other* badness that resulted from this case-insensitivity.

A Subversion problem: who was first responsible for an errant line?

May 2, 2010/stevereads/2 Comments

Suppose you want to find the first Subversion checkin where a particular string appeared. `svn blame [filename]` gets you some distance toward that goal, but it doesn’t entirely work: `svn blame` will tell you the person *most recently* responsible for tweaking the particular line of code where that string appears. If someone came along between when that line was introduced and now and, say, changed all Unix line endings to DOS ones, `svn blame` will suggest that the interloper is the one responsible for that line.

So what you want is the *first* revision number in which that change appeared. So far as I can tell, there’s no built-in svn command to give you this information. This shell-scripting business is the best I could come up with:

#!/bin/bash
string=$1
filename=$2

if test -z $string; then
echo “Must supply a string to search for”
exit 1
fi

if test -z $filename; then
echo “Must supply a filename to search”
exit 1
fi

# Get all svn revision numbers in which $filename
# was involved, in ascending order.
all_rev_nums=`svn log $filename
|grep -o ‘^r[0-9]+’
|grep -o ‘[0-9]+$’
|sort -n`

for revnum in $all_rev_nums; do
if svn cat -r$revnum $filename |grep -q $string; then
svn blame -r $revnum $filename |grep $string
# Since they’re in ascending order, we’ve found
# the first one. So we can quit now.
exit 0
fi
done

# If we never found $string in any revision
# of $filename, return an error.
exit 1

Tom Slee slaps down some new-economy nonsense

May 1, 2010April 21, 2026/stevereads/Leave a comment

Tom Slee is a fabulous author; his No One Makes You Shop At Wal-Mart is one of my favorite books of the last five years. He’s made a second career (which maybe he’ll turn into a book? I’d buy it) out of dispensing with a lot of new-economy nonsense; his latest salvo is against Clay Shirky. Shirky is a fine, provocative writer, but his love of technology leads him to some silly techno-idealism. Slee looks at the abstract structure of a Shirkian argument; it turns out that a lot of “Web 2.0”-inspired authors follow the same structure.

Slee’s series of responses to Chris Anderson’s “long tail” idea are in the same vein. I’d use the word “contrarian” for these, if I didn’t think that word had been sullied by Christopher Hitchens, and if I didn’t think it implied opposition for the sake of opposition. In a lot of the Web 2.0 nonsense, it would be hugely instructive for the Web 2.0 folks to be forced to argue the contrary of whatever it is that they’re claiming at that moment. Argue that “social media” won’t actually have any world-changing effect on anything. Argue that blockbusters will have just as much of a place in the 21st-century economy as they did in the 20th-century economy, and that those folks living on the long tail will have just as much trouble making a living as they ever have. Argue that the structure of a lot of economic processes is a classical arms race: my side adopts some new technology and temporarily moves ahead, but eventually your side does the same thing; the net effect is a wash. (You could have predicted, similarly, that even if sabermetrics was a valuable technology and initially helped teams with small budgets, that its value to those teams would eventually disappear as the Yankees caught wind of it.) Argue that while the Internet makes distributed teams more feasible and reduces transaction costs, and so might temporarily help small businesses, it will eventually be adopted by large companies as well. And so forth. I’d love to see the Shirkys of the world forced to write books arguing these positions with all the passion that they apply to their chosen viewpoints.

The trouble may be that the incentives are all wrong. It is much sexier to argue that some new flavor of the month will change the world than to argue that the world of the future will look a lot like the world of today. When everyone around you is swept up in talk of Twitter and FourSquare, you’re likely to do better if you assert with everyone else that these are the waves of the future. You will be invited to conferences; you will be asked to write books. Likewise, newspapers and policymakers will do a lot better if they talk about something sexy like terrorism than if they pledge to end 600,000 annual heart-disease deaths. The old and stable and known is boring, though it may well be true.

Perhaps Slee and I should team up and write a book about all this nonsense. Or maybe we should both sell out and write a book about how FourSquare Will Change Everything. It will sell reasonably well and earn us both a decent middle-class income, which we can then convert into a second book entitled Ha Ha We Were Just Kidding, Or: Your Latest Technology Idea Sucks.

Exporting an MP3 mix from iTunes

April 24, 2010/stevereads/Leave a comment

Suppose you make an MP3 mix for someone in iTunes. It’s easy enough to export the *list* of songs in that mix. And it’s easy enough to burn those songs to a CD. But it seems nontrivial to export *the actual MP3s*. Am I just not seeing an obvious way to do this?

Using a little shell-scripting, you can do this fairly easily:

#!/bin/bash
m3u_file=$1
export_dir=$2

function check_for_executables() {
# Paths to executables
TEST=/bin/test
MKDIR=/bin/mkdir
GREP=/usr/bin/grep
CP=/bin/cp
BASENAME=/usr/bin/basename

bad_execs=()
# Test that all executables are there
for executable in $TEST $MKDIR $GREP $CP $BASENAME; do
if ! $TEST -x $executable; then
bad_execs=”$bad_execs $executable”
fi
done

if ! $TEST -z $bad_execs; then
echo “Some required executables are missing: $bad_execs”
exit 1
fi
}
function process_args() {
if $TEST -z “$m3u_file”; then
echo “Must give the path to an M3U file.”
exit 1
fi

if $TEST -z $export_dir; then
export_dir=$(realpath $(pwd))
fi

echo “Exporting to $export_dir”

if ! $TEST -d $export_dir; then
echo “$export_dir does not exist or is not a directory; trying to create it”
if ! $MKDIR -p $export_dir; then
echo “Failed to create export directory $export_dir; quitting.”
exit 1
fi
fi
}
function export_playlist() {
i=0
cat “$m3u_file” |while read line; do
# Skip comment lines and blank lines
if echo $line |$GREP -q ‘^#|^s*$’; then
continue
fi
let i=i+1
if `echo $line |$GREP -q ‘^s*$’`; then
# line is blank, so skip it
continue
fi

filename=$(printf ‘%02d’ $i)_$($BASENAME “$line”)
export_path=${export_dir}/$filename
$CP “$line” “$export_path”
done
}

function main() {
check_for_executables
process_args
export_playlist
}

main

(Obvious improvements:

1. The `printf` statement prepends each track with its track number in the mix, and assumes that there are at most 100 songs in your mix. Have the script figure out the number of tracks in the mix, then compute the appropriate number of digits from that. The appropriate number of digits would be something like floor(log₁₀(n))+1, where n is the number of tracks in the mix and floor(x) is the greatest integer that’s no larger than x.
2. If the script is longer than about 10 lines, contains nontrivial logic, could usefully stand to compute logarithms, and can be meaningfully decomposed into functions, shell is no longer an appropriate language; use something like Perl or Python instead.

Exercises for the reader.)

Google to acquire my former employer for $1 billion?

April 21, 2010/stevereads/Leave a comment

If so, then holy fucking shit. Seriously.

A while back, I mentioned that Google was now in the business of giving multimodal travel directions: MBTA, say, to mass-transit system to mass-transit system to Amtrak. I mentioned that it would only be a matter of time before they’d connect up to airlines, etc.

What I didn’t mention there, but have always believed, is that the airline piece is something that only ITA could handle. They’ve been working on the problem of finding the cheapest flight between two cities for their entire history; if Google wanted to add airlines to its route-finding software, it would either have to reinvent what ITA did, or acquire ITA. Given that it took ITA the better part of a decade, and a team of the smartest people you could find, to solve this problem, it’s always been obvious that Google would acquire ITA rather than build this technology itself. If Google is planning to add air travel to its route-finding software, it follows that they’d have to acquire ITA.

ITA is sitting on the best kind of monopoly you can hope for: they’ve solved a problem that no one else can solve. They deserve any success that comes their way. And they can name their price if Google comes knocking. If Google decides not to buy, but they want to add air travel to their software, they’ll have to spend at least five years trying to do it. They’ll probably have to poach large numbers of ITA employees. They’ll need to hire people away from the airlines themselves. I’m no business strategist, but it certainly seems like ITA has them over a barrel here.

I, for one, bow deeply in the direction of ITA’s headquarters on Portland Street in Cambridge. If this works out the way it’s looking, then congratu-fucking-lations to you folks.

I’m still kind of in shock, even though this all makes perfect sense.

Because Google failed me for approximately the first time ever

April 17, 2010/stevereads/Leave a comment

…I would like to inform it that when I search for “preprandial beverage,” I expect it to give me the Wikipedia entry for
apéritif.

Kthxbai.

Health reform maybe striking a blow for gender equality?

April 8, 2010/stevereads/Leave a comment

This is really out of my butt, but I do wonder:

* A lot of couples would really like to split the childcare more evenly.
* It would really be ideal, toward that end, if both members of the partnership could work part-time and take care of the kids the other part of the time.
* But there’s very little meaningful part-time work in this country.
* Part-time work is made even less of an option because health insurance largely only goes to full-time employees.
* But under health reform, you’ll be able to get health insurance through an exchange, if your employer doesn’t offer it to you. (Note to self: look up the details of who’s eligible to buy on the exchanges.)
* So some couples won’t need to send one partner into full-time work, because they’ll be able to get health-insurance with only part-time labor.

Obviously this isn’t a full solution, and obviously there are benefits to full-time labor that part-time labor still won’t be able to match. But at the margin, at least, I suspect this will lead more than a few couples to split the childcare. Which is a good thing.

A self-taught master’s in CS

April 3, 2010/stevereads/Leave a comment

Perhaps this is a far-fetched idea, but let’s toss it out anyway: it occurs to me that I probably don’t have time in my schedule to get a master’s degree in computer science, but I’ve wanted for a long time to get one. I don’t have time because, as it is, I routinely work at my job until 8 or 9 at night. As I get more proficient at what I do, I expect I’ll work less, but at least for a while there’s just no way that I could do a job *and* get a master’s *and* be a reasonably good boyfriend *and* take regular trips up to New Hampshire to spend time with my girlfriend’s kids. Oh, *and* sleep.

So. That having been determined, I should spend the next few months/years building up master’s-level proficiency on my own. I should read books, watch videos, and write a lot of code on my own. A good master’s program will build a lot of theory as well, which means a lot of math. Historically, I’ve not been very good at learning math at an abstract level — but if I can code it up (e.g., writing a crypto algorithm), I can probably internalize it quickly.

Can anyone recommend a curriculum for my self-taught computer-science master’s degree? Recommend books? Recommend the sort of programming projects that one would encounter in a good master’s program?

Lewis Carroll, Alice’s Adventures in Wonderland and Through the Looking-Glass

April 3, 2010June 13, 2026/stevereads/Leave a comment

Cover of Alice's Adventures in Wonderland and Through the Looking-Glass, packaged together in one book published by Oxford. Cover is a painting of Alice looking at the queen and king of some suit that I forget; queen and king of cards, in any case.

These are books that I had inexplicably not read. I don’t know how that even happened. I certainly knew them by reputation, and I’d certainly known where many of the cultural allusions — the Red Queen running with all her might and yet still not moving, impossible things before breakfast, the Jabberwocky, etc., etc. — came from. There’s still lots of cleverness that I didn’t know about, like this bit:

“[…]The name of the song is called “HADDOCKS’ EYES.”‘

‘Oh, that’s the name of the song, is it?’ Alice said, trying to feel interested.

‘No, you don’t understand,’ the Knight said, looking a little vexed. ‘That’s what the name is CALLED. The name really IS “THE AGED AGED MAN.”‘

‘Then I ought to have said “That’s what the SONG is called”?’ Alice corrected herself.

‘No, you oughtn’t: that’s quite another thing! The SONG is called “WAYS AND MEANS”: but that’s only what it’s CALLED, you know!’

‘Well, what IS the song, then?’ said Alice, who was by this time completely bewildered.

‘I was coming to that,’ the Knight said. ‘The song really IS “A-SITTING ON A GATE”: and the tune’s my own invention.’

This is a terrific introduction to the use-mention distinction, I gather. Though normally when people talk about the use-mention distinction, they’ll differentiate between “Mount Everest” (a string containing 13 characters) and Mount Everest, a mountain in Asia. Here Carroll is making the split more decisive: the use is one thing (the song’s name), and the mention is entirely another (a string that doesn’t look at all like a representation of the song, if that makes sense). It’s clever, and not a little bewildering. I’d love to hear how much of it little Alice Liddell could process when Carroll was spinning his tales for her.

Nothing really happens in Carroll’s books. Alice ambles about, weird things happen, animals speak paradoxically to her, and that’s that. These are still very fun reads, though.

I love the packaging on the particular Oxford World’s Classics edition of Alice in Wonderland and Through the Looking-Glass that I read, but I think the endnotes are useless. Most often they point you to a reference that Carroll may have been making in the text, but most of the time I strongly doubted that he was making such a reference. Even if he had been, the endnotes pull you away from the text to make a point that does absolutely nothing to improve your understanding. It’s not as though Alice in Wonderland needs an exegesis on the level of Ulysses. So I’d advise buying this lovely edition, then ignoring the endnotes altogether.