27 Aug 2009
27Aug09
Does your test suite take too long (e.g. 5 minutes). Or did it and you solved it? Or it doesn’t but its getting worse?
Tell me more, I’d like to know 
Filed under: Uncategorized | 4 Comments
Does your test suite take too long (e.g. 5 minutes). Or did it and you solved it? Or it doesn’t but its getting worse?
Tell me more, I’d like to know
5 minutes for unit tests. 12 minutes for functional tests.
The effect of this is that developers run a subset of the tests before checking in, and buildbot is responsible for catching any unexpected regressions.
It would be nice if the tests took less time, but so far our half-hearted attempts to optimize them failed to discover any low-hanging fruit.
More numbers: on my development machine (Intel Core 2 Duo, 1.8 GHz) running just the unit tests:
Ran 6955 tests with 0 failures and 0 errors in 8 minutes 2.594 seconds.
That’s ~70 ms per test, on average. It would be interesting to compare this with numbers from other projects.
I’ve discussed bzr’s test suite duration. We’re sitting around 140ms per test, and many more tests.
We don’t have a solid separation of unit from integration tests: most of our tests act as integrated tests, with integrated costs.
Ours did (Drizzle) and we fixed it. How?
1) use transactions. Lots of disk Io sucks
2) use a ramdisk or libeatmydata to have sync() not really sync (amazingly, on a typical development workstation, where you don’t have all your RAM free as you also run firefox, libeatmydata is faster than a ram disk)
3) run things in parallel (you have N cpu cores, run at least N tests at once)