testrepository iteration for python projects

30Nov10

Tesetrepository has a really nice workflow for fixing a set of failing tests:

  1. Tell it about the failing tests (e.g. by doing a full test run, or running a single known failing test)
  2. Run just the known failing tests (testr run –failing)
  3. Make a change
  4. Goto step 2

As you fix up the tests testr will just give your test runner a smaller and smaller list of tests to run.

However I haven’t been able to use that feature when developing (most) Python programs.

Today though, I added the necessary support to testtools, and as a result subunit (which inherits its thin test runner shim from testtools) now supports –load-list. With this a simple .testr.conf can support this lovely workflow. This is the one used in testrepository itself: it runs the testrepository tests, which are regular unittest tests, using subunit.run – this gives it subunit output, and tells testrepository how to run a subset of tests.

[DEFAULT]
test_command=python -m subunit.run $IDOPTION testrepository.tests.test_suite
test_id_option=--load-list $IDFILE

About these ads


4 Responses to “testrepository iteration for python projects”

  1. You need to perform a full run to catch regressions introduced by your fix, that’s the whole point.

    • 2 rbtcollins

      To enlarge on Aaron’s response, consider a project with a 4 hour test suite, consisting of tens of thousands of individual tests.

      If you have 4 tests failing, and you run the entire test suite after each code change, you’ll only get *two* code changes done a day – and unless you’re a very very very good coder you’ll probably average under one correct code change a day.

      So there are two loops here:
      Outer loop: run all tests and identify any regressions
      Inner loop: change code, run tests that may fail due to the code change and only those tests.

      The test-runfailing-change cycle is an approximation to that ideal inner loop; in combination with code instrumentation (something I have a very crude prototype hanging around for) it should provide an efficient inner loop.

  2. 3 Aaron

    Thanks, Robert! This will be perfect for The Kraken.

    Aaron

  3. 4 Aaron

    Jon, doing a full test run is important, but the idea is that when you already have failing tests, you’ll want to fix them before doing your next full test run.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 877 other followers

%d bloggers like this: