Why upstreams should do distribution packaging

18Dec09

Software comes in many shapes and styles. One of the problems the author of software faces is distributing it to their users.

As distributors we should not discourage upstreams that wish to generate binary packages themselves, rather we should cooperate with them, and ideally they will end up maintaining their stable release packages in our distributions. Currently the Debian and Ubuntu communities have a tendancy to actively discourage this by objecting when an upstream software author includes a debian/ directory in their shipped code.  I don’t know if Redhat or Suse have similar concerns, but for the dpkg toolchain, the presence of an upstream debian directory can cause toolchain issues.

In this blog post, I hope to make a case that we should consider the toolchain issues bugs rather than just-the-way-it-is, or even features.

To start at the beginning, consider the difficulty of installing software: the harder it is to install a piece of software, the more important having it has to be for a user to jump through hoops to install it.

Thus projects which care about users will make it easy to install – and there is a spectrum of ease. At one end,

checkout from version control, install various build dependencies like autoconf, gcc and so on

through to

download and run this installer

Now, where some software authors get lucky, is when someone else makes it easy to install their software, they make binary packages, so that users can simply do

apt-get install product

Now some platforms like MacOSX and Microsoft Windows really do need an installer, but in the Unix world we generally have packaging systems that can track interdependencies between libraries, download needed dependencies automatically, perform uninstalls and so on. Binary packaging in a Linux distribution has numerous benefits including better management of security updates (because a binary package can sensibly use shared libraries that are not part of the LSB).

So given the above, its no surprise to me to see the following sort of discussion on #ubuntu-motu:

  1. upstream> Hi, I want to package product.
  2. developer> Hi, you should start by reading the packaging guide
  3. (upstream is understandably daunted – the packaging guide is a substantial amount of information, but only a small fraction is needed to package any one product.)

or (less usefully)

  1. upstream> Hi, I want to package product.
  2. developer> If you want to contribute, you should start with existing bugs
  3. upstream> But I want to package product.

Another conversation, which I think is very closely related is

  1. developer> Argh, product has a debian dir, why do they do this to me?!

The reasons for this should be pretty obvious at this point:

  • Folk want to make their product easy to install and are not themselves DD’s, DM’s or MOTU’s.
  • So they package it privately – such as in a PPA, or their own archive.
  • When they package it, they naturally put the packaging rules in their source tree.

Now, why should we encourage this, rather than ask the upstream to delete their debian directory?

Because it lets us, distributors, share the packaging effort with the upstream.

Upstreams that are making packages will likely be doing this for betas, or even daily builds. As such they will find issues related to new binaries, libraries and so on well in advance of their actual release. And if we are building on their efforts, rather than discarding them, we can spend less time repeating what they did and more packaging other things.

We can also encourage the upstream to become a maintainer in the distro and do their own uploads: many upstreams will come to this on their own, but by working with them as they take their early steps we can make this more likely and an easier path.

About these ads


14 Responses to “Why upstreams should do distribution packaging”

  1. Note that the dpkg-source v3 format entirely discards any debian/ directory shipped by upstream, which basically makes this problem go away.

  2. 2 pixelpapst

    Agreed, this upstream-unfriendly behavior on this specific front has been going on for far too long. Luckily, the introduction of the source v3 format (“3.0 (quilt)”, to be precise) has just made this problem a non-issue. So let’s encourage upstreams to use this format, and keep a debian/ dir in its version-control / tarballs. When Debian- or Ubuntu-specific changes are needed, we can upload the original tarball, a .debian.tar.gz, and dpkg-source -x will automatically replace the former’s debian/ dir with the contents of the latter.
    Then, regular communication with upstream should eventually lead to not only debian/patches/ to trickle back upstream, but also reorganizations of packaging, post-install scripts etc.
    The debian tarball should stay the place to implement debian packaging policy, but there is no reason upstream cannot make life easier for the rest of the .deb world by keeping around a debian/ dir. (Appearantly, even the upstream linux kernel can build .debs nowadays.)

  3. I agree with you and think we shouldn’t generally discourage the upstream from doing the Debian packaging work. I’m guilty of this, too. :) From my own experience it depends a lot on the upstream’s attitude. I have two upstreams who’s packages I just check and sponsor. They have a very good understanding of how to build proper Debian packages, they are subscribed to the BTS bug tracker and react to that. This is the (rare) perfect case.

    On the other hand I have dealt with a number of weird upstream developers. One case is where the software they build gets installed into /opt/somesoftware and they refuse to install their files elsewhere. Another case is an upstream developer who creates broken packages (having strange or no dependencies and which don’t build on more than one architecture) claiming that “it works for my users” and either Debian takes it as it is or leaves it. They often offer to download the packages from their own website and don’t care whether it works on the current stable version or not. They may even encourage end-users to upgrade to “unstable” which makes the user hate Debian in no time due to a broken system.

    And even if the upstream is friendly and helpful then I can fully understand if they stare at their screens at disbelief of the huge amount of documentation and knowledge that is needed to create Debian packages (policy, new maintainer’s guide, developer’s reference). To create proper Debian packages you need a lot of knowledge. The upstream developer is usually occupied with developing the actual software. I can hardly imagine how much time they would spend creating DEB packages, RPM packages and how many operating system installations they needed to test it.

    So if they show interest in maintaining the Debian package then we should help them to do it properly. Otherwise I’ll probably still just remove their debian/ directory, recreate the orig.tar.gz and debianize myself.

  4. As someone who is a Debian/Ubuntu package maintainer, and also an upstream author, I both agree and disagree with you.

    I understand how Debian and Ubuntu go together, and as such, I produce packaging for my software (and indeed the software my company produces) as a matter of course. However, I also understand that I know nothing about proper packaging for any of the RPM based distributions whatsoever.

    So, while I produce and provide spec files, they are not good and I’m not sure if I’d be more ashamed were I to tail to provide them, than I am ashamed of the quality of them as I provide them now. Unfortunately I lack the time and inclination to learn another four distributions’ particulars before packaging my software for them.

    I think upstreams being involved is a wonderful and laudible act which should be encouraged and promoted. However I also think that upstreams should *NOT* distribute their packaging as part of the upstream software tarball. Not least of which, I’d hate for Debian to have a new upload just because the Fedora packaging changed slightly for policy reasons.

  5. 5 Vadim P.

    Best solution I found out to go about it: Don’t give a damn. Package for Ubuntu in a PPA, and that’s all you need. If you can make a generic installer, even better.

    Forget about trying to get it into Ubuntu or Debian – because you’ll be wasting time creating the personal trust relationships necessary, or as you’ve described, be told to read the huge packaging manual yourself, which people do as their whole foss ‘job’.

    Hope someone interested will come about and package it. It works that way.

  6. Hi Robert,

    ok, when upstream wants to package their product for distributions, they should do this for every distribution on the market,

    Which means:

    1. RHEL/Fedora -> RPM
    2. OpenSuSE/SLES -> RPM + SuSE Specials
    3. Debian -> DEB
    4. Ubuntu -> DEB + Ubuntu Specials
    5. Gentoo -> eBuilds
    6. etc. pp.

    Means, Upstream needs to provide knowledge of at least those distributions, which most of upstream doesn’t have. Mostly they are using Fedora or OpenSuse and don’t have a clue about the others.
    Neither most upstream do have a clue about the tips and tricks of the packaging system.

    They should not, never ever, provide a .spec file for RPMs or a debian/ dir inside their distributable sourcecode tree.

    What they can do is to find people who are caring about upstream product and they can provide a special source tree for debian dirs or spec files for the different RPM distros.

    As far as I can say, Upstream does not know anything about distros and they don’t need to have…there are too many distros..upstreams should take care about their product to make it error free.

    Regards,

    \sh

  7. 7 Debian

    With dpkg-source v3 most of the reasons for us objecting against a debian/ dir in the source package go away because dpkg-source removes any debian/ dir from the unpacked source before unpacking the debian.tar.gz

  8. 8 Andrew

    Sorry, but I disagree. Sounds great for ubuntu! What about slackware users? fedora, arch, debian? Suse, mandirva, or any of the BSDs? Should upstream package all of them? Imagine the workload for the gnome project, if they had to package a gnome release for, say, 30 distributions.

  9. Your suggestion ignores the fact that while debian/* uses a standard set of files, the packaging implementation may not be reusable between different distibutions and releases. Either do tue different package naming conventions for dependencies, or just based on the building support used (cdbs, debhelper, version, etc).
    Keeping patches against a specific version of a debian/ dir does not help the “regular” packaging process.

  10. 10 Alberto

    I agree with the last part of your article, but just that part.

    Why?

    Because I think the “Waterfall incident” teach us that we need to assure that the software installed in Debian/Ubuntu is clear of any thread and the PPAs, although a help, are not the final answer. Nobody monitors what is in the PPAs.

    Besides that, there could be (and there are) problems of dependencies and incompatibilities because of software installed from the PPAs or other sources.

    I use PPAs all the time, but that is because I understand what they are: they are a repository to try the all brand new stuff, probably not prototypes as the daily repositories, but brand new, and that software has problems, like it needs, and updates, other software (software B), but a third software (software C) *needs* a lesser version and suddenly, the software *you need* to finish the work *you need* to deliver tomorrow doesn’t work any more (or worse, your PC doesn’t work at all).

    I can stand those problems, because I know that is part of the possibilities that the PC could be broken, but for most of the daily users, that is not a possibility and they begin to blame Debian/Ubuntu and says: They told me it was stable, but now doesn’t work, I prefer to came back to Windows (or Mac).

    So it’s better to let them to insert into the Debian/Ubuntu ecosystem, that to try to solve bugs derived from poor package management.

  11. 11 rbtcollins

    Wow, excellent responses. I’ll probably try to pull them altogether into a new post, but I’d like to make a couple of key notes – points I did not make as well as I would have liked to.

    Firstly, upstreams *do* distribution packaging. There is no need to ask them to or suggest that they should do it for e.g. gentoo. Not all upstreams do, and there is probably some interesting studies to do to determine what makes an upstream choose to start making packages. I think that an upstream which cares about usability/installability is a necessary condition. Upstreams that do package, do not package for every distribution out there: though as upstreams get bigger, one can be surprised :). What distributions an upstream cares about will be a factor of things like their user base, their preferred distribution, how much they care about long term supported distribution releases.

    Secondly, I’m not suggesting that we as distributors should say to every upstream ‘hey you should package for ‘. Rather I’m saying that when an upstream /chooses to package for /, we have a choice:
    * Collaborate – give them feedback, build on their work, work together.
    * Ignore their work – discard it without telling anyone.
    * Reject their work – ask them to stop doing what they are doing.

    Now its true that you can both collaborate and ask an upstream to put the packaging data outside their main tree – and this may be a good idea for some upstreams. Asking them to put it outside their main tree, and then ignoring it doesn’t count as collaboration!

  12. 12 toots

    Collaborating with upstream is a good thing, but I disagree with you when you state that upstream shouldn’t understand the packaging rules when preparing its package.

    Distributions have distro-specific concerns and policies and upstream should really be aware of that if he wants to provide a package in a correct shape.

    From my experience, lot of distro-specific things are not well understood by upstream, or even counter productive for him, such as, for instance, shipping internal version of libraries already packaged…

    • 13 rbtcollins

      I don’t think I claim that upstreams shouldn’t understand the rules; rather I claim that knowing all the rules is an unnecessary impediment: it is making the perfect the enemy of the good.

      And I totally agree that shipping internal versions of packaged libraries is bad – see my note about reusing shared libraries in this very post :). Part of the *cause* for that counter productive behaviour though, is the mindset of “we are delivering a product and library XYZ might not be available”. Working with distributions and understanding both our concerns (security updates, multiple versions of the same library getting loaded by different libraries in a single process, and even disk space size), and our solutions (package dependencies! FTW!) can only help address the issue.

  13. 14 Tommy.S

    Why I get the feeling that you have not understanded the upstream and downstream?

    Packagers are not upstream, they are downstream. Every distributor are downstream. The developers itself are on the upstream. Upstream is the place where the code is developed and from one upstream, multiple downstream’s gets their code and distribute it to their distros users.

    Upstream should NEVER need to do packaging. That is the job of of the downstream = distributors.
    The developers just maintain the upstream and develop it. Then distributors grab it, packages it and distributes it to their own users.

    Upstream (Maintainers/Development) -> Downstream (Distributors) -> Distribution users (Users/Clients)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 1,056 other followers

%d bloggers like this: