Zenhack.net

Test Code Is Still Code

26 Mar 2016

The Django tutorial has a section on testing, which contains one of the dumbest comments I’ve seen in a while:

It might seem that our tests are growing out of control […] the repetition is unaesthetic, compared to the elegant conciseness of the rest of our code.

It doesn’t matter. Let them grow. For the most part, you can write a test once and then forget about it.

Sometimes tests will need to be updated. Suppose that we amend our views so that only Questions with Choices are published. In that case, many of our existing tests will fail — telling us exactly which tests need to be amended to bring them up to date, so to that extent tests help look after themselves.

[…]

At worst, as you continue developing, you might find that you have some tests that are now redundant. Even that’s not a problem; in testing redundancy is a good thing.

As long as your tests are sensibly arranged, they won’t become unmanageable.

This reasoning sounded plausible to me, until I tried it, and saw where it got me. There’s a particular codebase I’ve worked on that for the most part was well organized, well documented, readable, maintainable code—except for the test suite.

The test suite covers around %40 of the code at this point. At one point it was more like 70%. Most of the test suite was written in an afternoon by one person, guided by exactly the kind of thinking described in the Django tutorial. The individual tests look a lot like the ones in that tutorial, and we basically took that approach for a while.

This was a mistake. Every time we made an architectural change in the software, changing the program itself was relatively simple, but the disruptions to the test suite were much more severe. We followed some rules that correspond to the bullet points about organization that the Django tutorial provides, but this will only get you so far. Code coverage got worse because folks didn’t want to wrangle with the test suite more than they had to, and while making changes to our nice, lean, well-thought-out piece of software was quick and easy, the test suite was one of those things where you never really felt comfortable making changes.

The notion that test code is different, and that redundancy and size are a non-issue is just wrong. Test code is still code, and all of the normal rules about how to write code apply. Having the tests did make us more confident in the correctness of our code, and of our changes, but it slowed us down in all of the ways that poorly maintained code always does.

As the test suite grew, it also became harder and harder to understand what was actually being tested. Coverage tools help, but they can’t tell you whether you’ve got good coverage of inputs, just lines of code. That’s just not good enough, and if you can’t understand your test suite, you’re in a really bad spot: You don’t want to delete it, since it is doing something important for you, but the lack of visibility makes it very hard to reason about how good your testing actually is.

So yeah. Worst advice ever. Unfortunately, the Django tutorial isn’t the first or only source I’ve heard it from. So, I’m saying something in the hopes that you won’t have to learn that the hard way. Here are some concrete lessons learned: