OpenStack Mitaka debrief

Well, last week was the 6-monthly OpenStack summit in Tokyo. It was fantastic to catch up with many folk, but with 5000 attendees, there are many more that I didn’t see than those that I did. Yet I find the sheer volume of face-to-face stuff nearly overwhelming. I wish it was quite a bit longer and less intense.

Over the next cycle I’ve committed to a few things…

  1. Kicking off TC leadership of scaling for OpenStack. That is, sparking the conversation with the broader community about what scaling means for us, and ensuring each project is paying some attention to it – in the same way that each project already pays attention to e.g. backwards compatibility – they can choose how much, and implementation and so on, but the basic user expectations and framework for thinking about it are shared across OpenStack. The performance working group is certainly related to this but scaling is different to performance.
  2. Replacing the oslo incubator process with one that creates the package straight away. This will go up as a spec for approval of course. The crux of the issue will be finding a way to preserve the freedom of early refactorings without API commitments, without breaking everything. The current approach in my head is to use versioned submodules within the package during the pre-1.0.0 phase, and liberally copy-paste things when API breaks are needed.
  3. Helping the app catalog folk a little bit by doing a review of their review guidelines – looking specifically for gaps (e.g. like the currently unsecured http attack vector).
  4. Start a broad discussion over changing the way we use minimum versions of requirements. Today we raise the minimum version of most requirements quite eagerly. Yet for some like libvirt we instead use feature detection and degrade gracefully when non-latest versions are installed. It seems likely that it would increase compatibility with distributions if we took that approach more widely, but we’d need some care to think through the ramifications.
  5. Kicking off a discussion about leadership training for TC & PTL members. We vote folk into these rolls, but leading isn’t a innate skill. With our constituency of over two thousand developers, spending some money on good leadership training seems like a sound investment. If the TC agrees that its a good idea, my plan is to seek funding from the Board, and aim to make the training be a pre-summit event. This was suggested to me by Colette Alexander.
  6. Seek some more eyeballs on the olso.messaging Kafka driver spec from the HP folk that have been working with Kafka.
  7. Establish connections between Yahoo & HP’s iLO team – they’re seeing the same sort of lockups we did with IPMI on the TripleO test cloud (and the infra-cloud folk are still seeing that) – so I want to see if we can get the bug fixed for everyone.
  8. Work up a clear spec on refactoring the testrepository and subunit2sql layers so that we have all the data store backends in one common repository, an HTTP REST API for consumers like openstack-health, and still have a good experience for CLI users.
  9. Lastly, but not least, work up a formal stabilisation cycle proposal to try and give everyone (product working group, users, core developers) what they want which we seem deadlocked on not doing today. The basic thing to me seems to be fear of the consequences of saying no to feature patches – for pretty good reason; many developers have their income directly tied to achieving things upstream, and when upstream says no, the ensuing discussion is fraught (and there is often information asymmetry present). What we probably need to do is find some balance point – and then socialise the plan very broadly – including the Board, so they can encourage member companies to look after their developers properly.

If any of these things are of interest to you, please feel free to reach out to me :).

Graceful introduction of test servers

A test server acts as a little RPC server where we can ask it to run some tests without paying a full new-process startup cost each time. They are a necessary precondition to online scheduling of tests (because without them the latency of scheduling a test will be orders of magnitude more time than executing the test), as well as potentially enabling better debugger glue by providing an explicit out of band interface.

It’s vital that we don’t break existing users of or testrepository when we bring this in – folk don’t react well to having their environment broken. Breaks could occur several different ways – but lets assume that an unmodified .testr.conf will not result in the server code being activated. (It would be nice in theory to Just Work and make things better, but there are lots of ways it could fail, starting with the fact that we have no negotiation step with the things we’re running, and anything else (e.g. exported environment variables) stands a high chance of being eaten by intermediaries like ssh, tox and so on).

So, assuming a new .testr.conf:

  1. A newer running with an old testrepository might drop into server mode and then not actually run any tests.
  2. A newer testrepository with an older might not go into server mode but not error cleanly.

Testr’s run command has two key interfaces with test backends. Firstly the list interface, where it queries for tests. This is only done when testr needs to know what tests exist (e.g. for offline scheduling). Secondly, the run interface where tests are executed.

In the server based world. testr will have one invoke-a-process interface, and that will offer the two existing interfaces over the basic RPC layer.

To avoid failure 1, we need to ensure we never ask for to go into server mode except when testrepository itself can handle it. That implies that we must not insert whatever change we are making into the run_command in .testr.conf, and instead either use a variable substition, or a whole new command key to configure it. I’m in favour of a new command key, because it places less constraints on implementors of other languages.

To avoid failure 2, we need to be able to rigorously determine if a process has gone into server mode. E.g. the server has to send a handshake command of some sort.

Lets talk about failure modes that can occur once we have .testr.conf configured and new and testrepository code.

  1. We might have version skew between releases of and testrepository on future updates to the RPC server.
  2. We might have a broken testr -> server channel
  3. We might have a broken server -> testr channel
  4. The server might go off into a busy loop or something

For 3, we should version the RPC protocol carefully so that any semantic differences can be detected. Obviously there is a tonne of prior art and everyone is going to scream ‘use grpc’ (or fav RPC of choice). Thats a very sensible thing to do, and subunit can actually sit on top of pretty arbitrary transports as long as they can handle bytestrings and timestamps. That said, my focus in this iteration is to enable the server, porting subunit’s transport to something else won’t save time there (because the RPC angle is going to be a tiny fraction of the development time). I think a simple (new, old) version scheme will do fine (think autotools library soname calculations). If testr offered (5, 1) it means it can speak all versions between 1 and 5, and as long as it speaks one of them, should pick that and use it. If we find the need to drop compatibility entirely with a version at some point, we raise the old to a version up from that and move on.

We can deal with 4 by pre-emptively sending a message from testr to the server – a hello message with the supported versions. Likewise, 5 can be dealt with by not considering the server ‘ok’ until we get its initial hello message with a chosen version.

If the server goes into a busy loop – I think we can largely ignore this for now, as its no different than today. (which is, the user notices, gets annoyed, and hits ctrl-C, or their CI job times out. Being able to discriminate between ‘the server is stuck’ and ‘a test is stuck’ would be good – and remembering that in a routed world we don’t know necessarily know the end point for any server…. it might just be routing.

What else – well one long running thing has been the desire to move away from requiring a clean stdin/stdout for test processes. Being broken when some test code decide to write to stdout is *not cool*. This new feature seems like an ideal time to address that. We can’t assume working networking (because e.g. tunnelling over ssh or a container console are important use cases). We could however write a little proxy that uses stdin/stdout with no test code, and then signals (however we’re doing that) that testr is listening on a local port, and tunnel it backwards. (If we choose something simple enough, it may even be possible to do that via parameterised ssh commands and no proxy at all). That does imply that testr itself still needs to be able to talk stdout/stdin. So – because testr has to keep doing that, I’m going to defer tackling this for now: it’s clearly scope creep and as such a dangerous temptation. Layer wise, it’s up to each server to decide how to be responsive when tests are cranky, and how to keep test output from compromising things. That does put the debugger integration work back (or at least, it leaves it as no better than the status quo) but its not in any way prejuidicial to it that I can tell.

Draft RPC spec

RPC packets will be stock subunit packets. Each packet will be for a test called ‘testrepository-rpc’ and contain a ‘application/json’ file attachment (with utf8 encoded text, per the default). The JSON message will be one of the messages defined below.

There are two endpoints, client (the initiator of the connection) and the server. Messages are not idempotent, and may be sent at any time from the client to the server. If a message requires a reply, the server may do so at any time, in any order. Subunit packets may be sent at any time from the client to the server, or the server to the client.

Overall lifecycle of a server:
  1. Client sends a Hello message.
  2. Server sends a Hello response.
  3. Both ends pick the highest common version to define future messages.
  4. Client sends commands, and server actions them.
  5. Client sends a Goodbye message.
  6. Server terminates itself.
Message definitions (version 1):
  • Hello

    Advises the peer of the protocol versions supported.
    {“msg”: “Hello”, “max”: 1, “min”: 1}

  • Goodbye

    Tells the server the client is finished and does not want to run any more tests. The server should cleanup and stop accepting messages. If the server was e.g. a trapdoor into a longer running process, it is undefined whether that longer running process should also terminate or not. No reply is permitted.
    {“msg”: “Goodbye”}

  • List

    Tells the server to list some tests. A “Done” reply is required after all the tests have been listed. The output from the command should be subunit “exists” packets describing the tests that the server can run that were listed in the message. The tests property is optional – if absent, list all available tests.
    {“msg”: “List”, “tests”: [“testid”, …], “nonce”: “arbitrary string here”}

  • Run

    Tells the server to run some tests. A “Done” reply is required after the tests have completed running. The output from the command should be a normal subunit stream resulting from running the tests specified. If the tests property is missing, run all available tests. Tests may be run in whatever order is most useful to the server.
    {“msg”: “List”, “tests”: [“testid”, …], “nonce”: “arbitrary string here”}

  • Done

    Tells the client that some requested command has completed. The nonce must be the nonce for the message that this is in reply to.
    {“msg”: “Done”, “nonce”: “arbitrary string here”}

Implementation sketch

The RPC protocol needs to be accessible to anyone doing this in Python, client *and* server, so subunit seems like the sensible place to define the protocol. It will be pure code – no IO interactions – along with sufficient feature work in subunit’s API to make glueing it into e.g. testrepository and straight forward.

In testr, we’ll look for a new command in .testr.conf, expressed much like the run command, and use that to determine that a server mode has been requested. If the server fails to start up, thats an error (e.g. it is up to users to get compatible code in place). When listing and running tests we’ll reuse the server except in isolation modes – both –isolated and –analyze-isolation – where reusing the server would violate the contract they have.

In, we’ll add a command line flag to opt-in to the server. In the first implementation, the server is going to just be in-line in the call stack ; no threads or anything. So each command will just be an API call within the existing testtools/unitest2 API with a single subunit packet tacked on the end. We may need to do some ugly stuff to get out of the stock run framework – but I think it is doable.

Testrepository roadmap 2015/16

Testrepository has been moderately successful – its very good at some of the things it aspired to (e.g. debugging sporadic test failures in parallel environments), but other angles have not really been explored.

I’ve set some time aside to correct this, in large part to facilitate some important features for tempest (which has its concurrency currently built on the meta-runner included in testrepository – and I’d like to enable the tempest authors to avoid having to write gnarly concurrency code :))

So my plan is to tackle a few things in the lead up to, and perhaps just after the Tokyo OpenStack summit. I wanted to socialise the proposed changes though, and thus this blog post.


Firstly, a long standing issue is that when one tests several different configurations, testrepository is poor at reporting failures that are configuration specific. For instance, imagine that your test suite is run with both Python 2.7 and 3.4, and both results are loaded into your repository. If a given test ‘X’ fails in the first run, and not the second… after the second run is loaded, it will be reported as ‘passing’.

My proposed fix for this is to call the name of each such run a ‘profile’ and use tags to differentiate between the two runs. So you’d tag the 2.7 run perhaps ‘py27’ and the second ‘py34’, and then tell testrepository that the ‘py27’ and ‘py34’ tags are being used to identify profiles. After that testrepository will only consider two test to apply to the same test if the tags match. Tags that are not specified as being for profiles (e.g. the worker-N tags that the testrepository runner adds to track backends that tests run in) won’t be considered in that comparison. This well then allow testrepository to track that each run was separate and the results are not meant to replace each other. The use of tags allows for test matrices too, in principle– consider python version as one dimension, operating system version as another, and database engine as a third — it would be up to the user. I don’t plan to directly implement a matrix system in the first iteration. A different, more dynamic model is in principle possible: don’t tag things, just log events that will give clues and correlate later – thats not precluded by this tag based approach, and we can always add such a thing later.

The output for the queries of the datastore need to be updated though – we don’t currently report tags in e.g. ‘testr failing –list’. This is a little tricky: the listing format is intended to be a mix of nice-for-humans, and machine consumption. Another approach we considered was to namespace the tests with the profile. This has a couple of disadvantages: it may break an unknown number of deployments if the chosen separator is already in use by people, and secondly, it mixes structured and free-form data in a lossy way. One example of that would be that we’d start interpreting all test ids to see if they are – or are not – namespaced with a profile : thats likely to be fragile, at best. On the other hand it would very easily fit into the list format – which is why it was appealing. On balance though, the fragility and conflation would just add technical debt. Instead, we’ll do the following:

  1. Anything that needs to output a flat list of tests will output that for just one profile. An option will be added to allow querying the profiles for which results might be given. The default will start erroring with a list of available profiles if more than one profile has been specified.
  2. We’ll define a minimal JSON schema for reporting multiple profiles in such places. The excellent jq tool can be used to manipulate that in shell command lines. A command line option will opt into receiving this.

Testrepository has two very related programs inside itself. There is the data store and the various queries it can do – e.g. ‘testr load’ and ‘testr failing’. Then there is the meta-runner, which knows how to run some test processes to execute tests. While strictly speaking this is optional, its been very convenient for working with Python tests to have the meta-runner connected to testr and able to do in-process querying.

The meta-runner will benefit from being updated as well. My intent is to make it capable of running all the tests from all the profiles the user specifies, storing that as one single run in the datastore. Two commands in particular need to change here – `testr list-tests` needs to change in line with the test listing above, and `testr run –load-list` needs to be taught how to deal with multiple profiles. I plan to add a command line option to tell it that JSON is being used, and to select tests across all profiles when a simple list or a test regex is given. Finally the command line can benefit from a command line option to select one or more profiles.


The meta-runner has a crude scheduler – it balances based on historic performance prior to running any backend. An online scheduler will give much greater performance in both unseeded, and skewed data cases- e.g.if many long tests fail due to a bug the run after that will often have some workers finishing well before others – leading to slow test times.

The plan here is to finish the implementation of bidirectional channels to test backends, and then dispatch work to them incrementally

Concurrency plans

Tempest wants to be able to run some tests completely independently, and then others can run together arbitrarily. To facilitate this, the online scheduler will be extended to permit describing an overall plan to run through – e.g. a list of segments, where each segment describes one or more tests that can be run together. The UI to supply that to the scheduler will probably start out as a JSON file listing exact test ids and we can iterate from there based on their experience.

The merits of (careful) impatience

The Python packaging ecosystem has long desired a overhaul and implementation of designed features, but it often stalls on adoption.

I think its time to propose a guiding principle for incremental change.

be carefully impatient

The cautious approach for delivering a new feature in the infrastructure looks like this:

  1. Design the change.
  2. Implement the change(s) needed, in a new major version (e.g Metadata-2.0).
  3. Wait for the new version to be the default everywhere.
  4. Tell users they can use it.

This is frankly terrible. Firstly, we cannot really identify ‘default everywhere’. We can identify ‘default in known distributions’, but behind the firewall setups may lag arbitrarily far behind. Secondly, it makes the cycle time for getting user feedback extraordinarily long: decade plus time windows. Thirdly, as a consequence, we run a large risk of running ahead of our users and delivering less good fixes and improvements than we might do if they were using our latest things and giving us feedback.

So here is how I think we should deliver things instead:

  1. Design the change with specific care that it fails closed and is opt-in.
  2. Implement the change(s) needed, in a new minor version of the tools.
  3. Tell users they can use it.

So, why do I think we can skip waiting for it to be a default?

pip, wheel and setuptools are just as able to be updated as any other Python component. If someone is installing (say) numpy via pip (or easy-install), then by definition they are willing to use things from PyPI, and pip and setuptools are in that category.

And if they are not installing via pip, then the Python packaging ecosystem does not affect them.

If we have opt-in as a design principle, then the adoption process will be bottom up: projects that are willing to say to their users ‘you need new versions of pip and setuptools’ can do so, and use the feature immediately. Projects that want to support users installing their packages with pip but aren’t willing to ask that they also upgrade their pip can hold off.

If we have fails-closed as a design principle, then when a project has opted in, and the user installing the package hasn’t upgraded their pip, things will at least fail rather than silently doing the wrong thing.

I had experience of this in Mock recently: the 1.1.0 and up releases depended on setuptools 17.1. The minimum setuptools we could have depended on (while publishing wheels) was still newer than that in Ubuntu Precise (not to mention RHEL!), so we were forcing an upgrade regardless.

This worked ok but we had two significant issues. Firstly, folk with incorrect Python paths can end up shadowing system installed packages, and for some reason ‘six’ triggered this for multiple users. Secondly, we had a number of different attempts to clearly signal the dependency, as the new features we were using did not fail closed: they were silently ignored by sufficiently old setuptools.

We ended up with a setup_requires="setuptools>17.1" clause in, which we’re hopeful will fail, or Just Work, consistently.

Bootstrapping developer environments for OpenStack

When I first joined OpenStack, it was incredibly hard to get a development environment working. Its better now in some ways, but not in others.

There are three key things.

Firstly, if you’re going to run devstack (or equally tempest-on-devstack), make a dedicated VM to do it. Use whatever hypervisor you want, but don’t you dare run it on baremetal. Yes it can work on baremetal, but you’ll spend way more time learning about quirks than getting stuff done. Once you’re familiar with devstack you can make an educated choice.

Secondly, if you’re not running devstack [e.g. you’re just hacking on an oslo library or gertty or something] there are a bunch of system packages that will need installing to let various Python packages compile. This was one of the places that was hardest to get going on when I started, and still is pretty hard. I started the bindep project to address this, and its in the process of being rolled out now within OpenStack projects. If the project you’re hacking on has an other-requirements.txt file, then you can just do sudo apt-get install $(bindep -b) – but you’ll need bindep installed first, and its not in distros yet. If there is no such file there is a master fallback one which brings in everything maintained by the OpenStack Infra team. Download it somewhere and add the -f bindep-fallback.txt to your bindep call.

Which brings me to the last point. How do you get bootstrapped with bindep and tox and all the other things that you’ll need. The OpenStack developer docs tell you to install system-wide using apt-get install, and then system-wide install tox. This usually works, but isn’t suitable for multi-user machines, and some distributions don’t include enough metadata for pip to do the right thing, and its confusing to have two different systems managing the same packages.

I prefer to use virtualenv directly, with no reliance on system packages other than truely global things like Python itself. virtualenv is a lovely thing, and it bundles three other important Python tools – pip, wheel and setuptools. All combined this will let you get a fully up to date development environment going without root access – and without confusing your package manager. I still use it even with Python versions that include the venv module, because having the bundled components be up to date is important.

With a virtualenv to host tox and bindep and any other globally-relevant tools, you can run those tools without interacting with your system package manager at all[except when you finally remove the version of Python they use from the system].

Here is how I accomplish this.

  1. Start with Python [any version (2.7 or newer)] installed.
  2. pick a place to make the virtualenv. E.g. ~/devtools
  3. Get a copy of virtualenv to run locally from source. 13.1.0 is a good safe version (at time of writing).
  4. cd into virtualenv-13.1.0 (or whatever number)
  5. Run python ./ ~/devtools
  6. Activate the virtualenv
  7. Install bindep and tox and anything else you need globally using pip. The pip in the virtualenv will be nice and fresh so you’ll get full caching and so on.
  8. (optional) symlink those tools into your user specific path. E.g. from ~/devtools/bin to ~/bin, or some similar directory.
    There are two key gotchas here. Firstly, if you use a bind-dir mounted home in containers, you won’t want this in your user path, since venvs are not very portable. Instead make a per-container path directory you can symlink into.
    Secondly, there is cruft in ~/devtools/bin, so don’t even for a second think of just adding it to your PATH: you’ll end up running the python within the virtualenv by mistake at some point and deeply regretting it.
    If you don’t do this, then just activate the virtualenv any time you want to use the tools.

And now you can circle back, use bindep to install binary dependencies, and off you go!

Revisiting the Fixture API – handling leaky resources

Fixtures are one of the innovations I’m most happy with.

A Fixture is an enhanced context manager. The enhancements are:

  • There’s an API for gathering debugging information from the fixture (rather than depending on side effects such as the logging module or stdout). This makes it easy to attach log files from servers (for instance rabbitfixture does this).
  • There is glue to support composing other fixtures while still exposing errors from any fixture in the composed set.

OpenStack’s Neutron has been using fixtures in its test suite for some time, but is finding that writing correct fixtures is hard. In particular, they were leaking processes when a fixture would fail during setUp / __enter__ – and then not be cleaned up by the testtools / fixtures useFixture function.

There are several things we can do to improve the situation.

  • We could make the convenience APIs like useFixture add a try:/finally: and call cleanUp() when setUp fails. This involves making cleanUp() be callable in more situations than it is today.
  • We could make setUp itself do that, advising users to override a different function; this would hide the failure interactions internally, but wouldn’t benefit existing fixtures until they are rewritten to not override setUp.
  • We could provide a decorator that folk with fragile setUp’s (e.g. those that involve IO) could use to robustify their fixtures.

The highest leverage change is the first, but is it safe and suitable? Lets look at PEP-343.

In PEP-343 we see the following translation of with expressions:

with EXPR as VAR:
mgr = (EXPR)
exit = type(mgr).__exit__
value = type(mgr).__enter__(mgr)
exc = True
        VAR = value
        exc = False
        if not exit(mgr, *sys.exc_info()):
    if exc:
        exit(mgr, None, None, None)

This means that using a Fixture which may leak external resources when setUp fails is unsafe via with. Therefore we can’t use the first solution.

Decorators are nice, but somewhat noisy and opt-in. Both decorators and a different setUp in the base class will require extending the protocol to specify when cleanUp can be called more precisely.

If we make the documentation advise users to override a specific method, and setUp does this in the event of failure, I think we’ll have somewhat more uptake. So – thats the route I’m going to head down.

There’s one more thing to consider, which is access to debugging information of failures in setUp. Since the object will have been cleaned up, accessing logs etc will be hard. I think if we raise an additional exception into the MultiException with the details objects, it will be possible for fixtures to provide those details, though they will need buffering in memory (or some sophisticated lazy-delete logic such as holding a reference to an unlinked fd).

Improving dependency handling upstream (for openstack)

This is, in part, a follow up to my post a few weeks ago.

I want to touch on the things we need to improve to have robust plumbing supporting openstack’s CI and devstack needs.


We want to be able to use ‘extras’ to declare the dependencies needed for different backends. This is a setuptools requirement syntax where a project can advertise additional dependencies for different use cases, which users (or other depending projects) can then trigger using '[]'. E.g. 'pip install requests[security]' says ‘install requests and the additional ‘security’ extras. We don’t know yet whether we will use 'nova[mysql]' or 'nova oslo.db[mysql]', but something like that. To use this we need to:

  1. teach pbr about reflecting requirements into the 'extras_require' keyword to setup (because while setuptools supports it in, we want a constant value with everything about individual projects declarative).  James Polley has a patch for pbr.
  2. Fix pip to handle 'pip install ./nova[mysql]'. This is issue 1236 – which has an open PR that may fix it. We should help review and test it.

Testing different setups may well need a similar facility, but its not clear yet how to best express that. We may need to standardise on using an extra called 'test' and just ensure our tox.ini knows to install that. That would be nice anyway, to get away from having to know about 'test_requirements.txt'.

pip dependency resolution

Currently pip has a very straight forward resolution algorithm: Only user supplied requirements can conflict at all, and the first mention of any distribution causes a distribution to be selected that matches that mention – all other mentions are simply ignored. This is issue 988, and its one of a cluster that affect OpenStack. The impact on OpenStack is that we have things install ok with pip, and then break in CI, because an incompatible version is installed. I have a patch up for this. Early adopters solicited!

incremental installations need dependency resolution

Say you’ve installed Neutron, which depends on oslo.db >=1.10. And you then install an older Nova which depends on oslo.db <1.10. What should happen? Ideally an error in this case, because the requirements are disjoint. And if they do overlap, the installed version should be adjusted to be compatible. Right now, no error occurs and oslo.db will be downgraded breaking Neutron. This is pip issue 2687. Currently no-one is working on this, and since it requires dependency resolution, fixing 988 first makes a lot of sense. It should be possible to at least make things error with a much more shallow patch though, if someone wished to work on it right now – or you could build on top of my resolver branch. This has also been a cause of numerous CI failures when we do releases, typically right around the time the servers branch. One thing that might be nice for us, since we know a full set of working packages, is to be able to say upfront to pip what versions are compatible, and then let only the needed things be brought in. pip issue 2731

PEP-426 environment markers need polish

PEP-426 introduced a micro-language for describing the situations when a particular dependency applies. For instance, to use argparse on Python < 2.7, you can say "python_version<'2.7'" as a marker for the argparse entry in your requirements. But there are some rough edges.

  • Some comparison operators are missing.
  • The documentation and user guidance needs improvement.
  • Environment markers can’t be used inside individual requirements, only as a filter on extra_requires. To express the argparse example above today (using a working operator), you need to pass the following to setup().
    extra_requires={':python_version=="2.6"': ['argparse']}

    It would be more straightforward to permit the syntax pip supports, where each requirement can be annotated with a marker.


    This might be setuptools issue 353.

  • pbr doesn’t reflect environment markers from its input files (requirements.txt etc) into setup keyword argument. James Polley has a patch for this (the same one enabling extras support in setup.cfg).

pip handling setup_requires

We run into setup_requires in two places in OpenStack; firstly we use that ourselves for pbr, but to avoid triggering easy_install we manually install pbr everywhere ourselves. Secondly, projects that are in the transitive dependencies of OpenStack use setup_requires, and we end up triggering easy_install for them. easy_install is a concern for us because of the decreased reliability and issues with corporate egress firewalls, and its security is not as robust as pips – and there’s no reason it should be, with pip being such a good tool.

However pip can’t handle setup_requires today. Doing so requires changes to setuptools and to pip.

  • setuptools needs some way to report to pip what the setup_requires are without triggering easy_install. Ronny Pfannschmidt has mentioned he may be working on this, but I’m not sure if there is a patch ready or not. A possible further enhancement would be to put the setup_requires in setup.cfg in a totally declarative fashion, but this may require environment marker support first, since the current procedural approach is very flexible and can take Python version and platform into account.
  • pip needs to be able to temporarily put things that it won’t be installing into the PYTHONPATH for packages it is building. The current internals are not suited for this (the target and source and needs of requirements being downloaded are all confounded). However once my resolver patch lands, there will be a nice cache layer that can deliver a ready-to-install directory for any requirement, which should make a simple recursive implementation quite reasonable. The resolver work will probably need further refactoring to make the resolver be decoupled from the user supplied requirements, but compared to the ground already covered, that should be straight forward. One thing folk tackling this should be aware of is an open question around location requirements. Say someone is installing foo from a git repository. And foo is also a setup requirement of some other package bar being installed at the same time. Should that foo from git be used for the setup of bar? I’m not sure of the answer (what if the version of foo is incompatible with version bar needs?) – but one is needed :).

So thats about it – if you’re interested in helping the plumbing that supports OpenStacks CI and devstack systems, please pick one of these issues and help out. Test patches, review code, write a patch, or just tell me why we don’t need to do something :)