The Django tutorial has a section on testing, which contains one of the dumbest comments I’ve seen in a while:
It might seem that our tests are growing out of control […] the repetition is unaesthetic, compared to the elegant conciseness of the rest of our code.
It doesn’t matter. Let them grow. For the most part, you can write a test once and then forget about it.
Sometimes tests will need to be updated. Suppose that we amend our views so that only Questions with Choices are published. In that case, many of our existing tests will fail — telling us exactly which tests need to be amended to bring them up to date, so to that extent tests help look after themselves.
[…]
At worst, as you continue developing, you might find that you have some tests that are now redundant. Even that’s not a problem; in testing redundancy is a good thing.
As long as your tests are sensibly arranged, they won’t become unmanageable.
This reasoning sounded plausible to me, until I tried it, and saw where it got me. There’s a particular codebase I’ve worked on that for the most part was well organized, well documented, readable, maintainable code—except for the test suite.
The test suite covers around %40 of the code at this point. At one point it was more like 70%. Most of the test suite was written in an afternoon by one person, guided by exactly the kind of thinking described in the Django tutorial. The individual tests look a lot like the ones in that tutorial, and we basically took that approach for a while.
This was a mistake. Every time we made an architectural change in the software, changing the program itself was relatively simple, but the disruptions to the test suite were much more severe. We followed some rules that correspond to the bullet points about organization that the Django tutorial provides, but this will only get you so far. Code coverage got worse because folks didn’t want to wrangle with the test suite more than they had to, and while making changes to our nice, lean, well-thought-out piece of software was quick and easy, the test suite was one of those things where you never really felt comfortable making changes.
The notion that test code is different, and that redundancy and size are a non-issue is just wrong. Test code is still code, and all of the normal rules about how to write code apply. Having the tests did make us more confident in the correctness of our code, and of our changes, but it slowed us down in all of the ways that poorly maintained code always does.
As the test suite grew, it also became harder and harder to understand what was actually being tested. Coverage tools help, but they can’t tell you whether you’ve got good coverage of inputs, just lines of code. That’s just not good enough, and if you can’t understand your test suite, you’re in a really bad spot: You don’t want to delete it, since it is doing something important for you, but the lack of visibility makes it very hard to reason about how good your testing actually is.
So yeah. Worst advice ever. Unfortunately, the Django tutorial isn’t the first or only source I’ve heard it from. So, I’m saying something in the hopes that you won’t have to learn that the hard way. Here are some concrete lessons learned:
- Test code is still code. If your programmer instincts are telling you something is wrong, you should listen just as you would with any other piece of software.
- Don’t Repeat Yourself. Standard engineering principle, but one of the most frequently ignored when writing tests. Put common code in subroutines/fixtures/etc.
- Think about design. With the test suite I discussed above, we had a similar structure where there was a database, and most of the tests were checking things that needed the database to be in a certain state to make sense (contain particular objects, for example). These were slightly different for each test, so looking at the tests didn’t reveal obvious chunks of code that could just be moved to a fixture. However, having the setup code in every test made them a lot harder to read. Often the check itself was a one-line assert statement, but there were four or five lines of code putting things in the database first. Later on we took the approach of having a fixture that populates the database with a pretty large number of objects — enough that most tests could find the objects they need. This makes it a lot easier to write tests whose function is obvious by looking at them.
- Make your test suite understandable. You should be able to look through the test suite and have a good idea of what is being tested and what isn’t. Coverage tools are only a rough heuristic, and there’s no substitute for being able to understand what the tests are doing for you.
- Make sure your tests are easy to modify. Having a good test suite can embolden you to make big changes, knowing there’s a safety net to catch you when you fall, but if your test suite becomes hard to rework, that can actually prevent you from making changes.
- Just because a test fails when it needs modification, does not mean modifying it (and others like it) is trivial. You can end up changing your architecture in a way that requires similar rethinking of your tests. Even if there’s not much concern about making mistakes, it can still be a lot of work.