Tesetrepository has a really nice workflow for fixing a set of failing tests:
- Tell it about the failing tests (e.g. by doing a full test run, or running a single known failing test)
- Run just the known failing tests (testr run –failing)
- Make a change
- Goto step 2
As you fix up the tests testr will just give your test runner a smaller and smaller list of tests to run.
However I haven’t been able to use that feature when developing (most) Python programs.
Today though, I added the necessary support to testtools, and as a result subunit (which inherits its thin test runner shim from testtools) now supports –load-list. With this a simple .testr.conf can support this lovely workflow. This is the one used in testrepository itself: it runs the testrepository tests, which are regular unittest tests, using subunit.run – this gives it subunit output, and tells testrepository how to run a subset of tests.
[DEFAULT]
test_command=python -m subunit.run $IDOPTION testrepository.tests.test_suite
test_id_option=--load-list $IDFILE
You need to perform a full run to catch regressions introduced by your fix, that’s the whole point.
To enlarge on Aaron’s response, consider a project with a 4 hour test suite, consisting of tens of thousands of individual tests.
If you have 4 tests failing, and you run the entire test suite after each code change, you’ll only get *two* code changes done a day – and unless you’re a very very very good coder you’ll probably average under one correct code change a day.
So there are two loops here:
Outer loop: run all tests and identify any regressions
Inner loop: change code, run tests that may fail due to the code change and only those tests.
The test-runfailing-change cycle is an approximation to that ideal inner loop; in combination with code instrumentation (something I have a very crude prototype hanging around for) it should provide an efficient inner loop.
Thanks, Robert! This will be perfect for The Kraken.
Aaron
Jon, doing a full test run is important, but the idea is that when you already have failing tests, you’ll want to fix them before doing your next full test run.