So a while back I blogged about maintainable test suites. One of the things I’ve been doing since is fiddling with the heart of the fixtures concept.
To refresh your memory, I’m defining fixture as some basic state you want to reach as part of doing a test. For instance, when you’ve mocked out 2 system calls in preparation for some test code – that represent a state you want to reach. When you’ve loaded sample data into a database before running the actual code you want to make assertions about – that also represents a state you want to reach. So does simply combining three or four objects so you can run some code.
Now, there are existing frameworks in python for this sort of thing. testresources and testscenarios both go some way towards this (and I and to blame for them :)), so does the zope testrunner with layers, and the testfixtures project has some lovely stuff as well. And this is without even mentioning py.test!
There are a few things that you need from the point of view of running a test and establishing this state:
- You need to be able to describe the state (e.g. using python code) that you wish to achieve.
- The test framework needs to be able to put that state into place when running the test. (And not before because that might interfere with other tests)
- And the state needs to be able to be cleaned up.
Large test suites or test suites dealing with various sorts of external facilities will also often want to optimise this process and put the same state into place for many tests. The (and I’m not exaggerating) terrible setUpClass and setUpModule and other similar helpers are often abused for this.
Why are they terrible? They are terrible because they are fragile; there is no (defined in the contract) way to check that the state is valid for the next test, and its common to see false passes and false failures in tests using setUpClass and similar.
So we also need some way to reuse such expensive things while still having a way to check that test isolation hasn’t been compromised.
Having looked around, I’ve come to the conclusion we’ll all benefit if there is a single core protocol for doing these things, something that can be used and built on in many different ways for many different purposes. There was nothing (that I found) that actually met all these requires and was also tasteful enough that folk might really like using it.
I give you ‘fixtures‘. Or on Launchpad. This small API is intended to be a common contract that all sorts of different higher level test libraries can build on. As such it has little to no policy or syntatic sugar.
It does have a nice core, integration with pyunit.TestCase, and I’m going to add a library of useful generic fixtures (like temporary directories, environment isolators and so on) to it. I’d be delighted to add more committers to the project, and intend to have it be both Python 2.x and 3.x compatible (if its not already – my CI machine isn’t back online after the move yet, I’m short of round tuits).
Now, if you’re writing some code like:
class MyTest(TestCase):
def setUp(self):
foo = Foo()
bar = Bar()
self.quux = Quux(Foo(), Bar())
self.addCleanup(self.quux.done)
You can make it reusable across your code base simply by moving it into a fixture like this:
class QuuxFixture(fixtures.Fixture):
def setUp(self):
foo = Foo()
bar = Bar()
self.quux = Quux(Foo(), Bar())
self.addCleanup(self.quux.done)
class MyTest(TestCase, fixtures.TestWithFixtures):
def setUp(self):
self.useFixture(QuuxFixture)
I do hope that the major frameworks (nose, py.test, unittest2, twisted) will include the useFixture glue themselves shortly; I will offer it as a patch to the code after giving it some time to settle. Further possibilities include declared fixtures for tests, and we should be able to make setUpClass better by letting fixtures installed during it get reset between tests.
Code happens 