• Immutable Page
• Info
• Attachments
• More Actions: Raw Text Print View Render as Docbook Delete Cache ------------------------ Check Spelling Like Pages Local Site Map ------------------------ Rename Page Delete Page ------------------------ Subscribe User ------------------------ Remove Spam Revert to this revision Package Pages Sync Pages ------------------------ Load Save SlideShow

Non-Native Packages

We want to be able to produce non-native packages sometimes with the BuildBranchToArchive feature. Currently we just produce native packages to sidestep the issues.

A non-native package is one that has an "upstream" distinct from the distribution - the vast majority of packages. In this case the format of the source package separates the upstream tarball from the packaging changes made from the distribution. This has a few benefits:

• Storage and transmission costs are lower as the tarball can be reused if it doesn't change between two uploads.
• Clear separation of the packaging changes can help with auditing those changes.
• Using a tarball provided by upstream allows you to compare checksums to see if changes have been introduces in the upstream tarball. This is useful as the upstream tarball isn't usually audited when uploaded to the distribution, and just trusted, so could be a vector for introducing malignant code. If the checksum matches the upstream one then no code can have been introduced by the packager.

A native package is different in that all the code, upstream or otherwise, is just stored in a single tarball in the source package. This is usually used where there is no upstream that is distinct from the distribution, but is not limited to this use case.

While it's always possible to just create a native package, we would miss out on the advantages listed above. In particular it is crucial for Ubuntu packages, and so this feature is a blocker for Ubuntu adoption.

All that is required to build non-native packages is to provide an appropriate tarball when we build the source package. bzr-builder will take care of these details, so we just need to be concerned with getting that tarball on to the build slave.

Use cases

1. Eve wants to build a package to the Ubuntu archive and re-use the tarball that is already there. She wants to ensure that a native package is not created unintentionally.
2. Phil builds to his PPA and it also re-uses the tarball from the main archive. He doesn't care too much if the package ends up being native.
3. Fabien does daily builds and produces a new upstream version every day, and so tarballs can't be re-used, but may want to be created.

Constraints

We have to stay within the rules of packaging as known in Debian/Ubuntu.

1. tarballs are named after the upstream version.
2. Once a tarball of a particular tarball is accepted in to an archive, it can't be changed or replaced.
3. Uploading a tarball claiming to be a particular version when it isn't is wrong.
4. Subsequent uploads of the same upstream version must use the the exact same upstream tarball due to #2, (unless the compression format changes, but there can only be one tarball of any particular compression format for one upstream version)

Cases

1. New upstream version of a package
2. Subsequent upload of the same upstream version
3. Native package

Design

What we have to do is produce a tarball, either from the target archive, or the main archive if it is a PPA and the PPA itself doesn't have the right file. That tarball is named after the upstream version that we are going to produce.

There are three ways we can do this:

1. Send the tarball to the slave and have it placed in the right place.
2. Have the slave download the tarball from the librarian.
3. Use pristine-tar information in the branch to recreate the tarball.

The third option is the best, as it allows anyone with the branch to do it, rather than just a build slave with the machinery that 1 or 2 would provide.

Therefore we have a constraint that the branch you build from must have the pristine-tar information for the tarball you want to produce in order to get a non-native package. Let's see how that affects our use cases:

1. The distro developers should be using merge-upstream to merge new upstream versions, which will store the pristine-tar information. We can also have a rule that states if the package isn't declared as native in the bzr-builddeb config then it must have the pristine tar information for the version being produced.
2. Phil can use merge-upstream if he likes, and if he builds on top of the distro's branches he will have this. If he doesn't care then he may want to turn off the pristine-tar requirement, so that could be a per-archive setting, but we could just enforce that requirement for now at least.
3. Daily builds are in essence producing a new upstream snapshot every day, meaning that re-using tarballs has less benefit over time. With the scheme outlined above two things will happen:
   • If the upstream part of the version number changes every day then it would fail, unless the native checking was disabled.
   • If the upstream part stays the same then it would work using the tarball from the distro as above.

This leaves us with one particularly interesting case, where you are doing a daily build, but the upstream version changes every day. You can't use the last upstream tarball you have and just rename it, as that breaks constraint #3. Therefore we have two options:

1. Produce native packages as we do now.
2. Create a tarball from the branches we use.

They are approximately equal in terms of space savings, with #2 having the opportunity to nudge it if the upstream version number doesn't change on some days. The second option is more difficult to implement though, so I would suggest #1.

That just leaves us requiring a way to disable the check that fails if it is not a native package and has no pristine-tar data for some archives (just (some) daily builds, or some daily builds and some other archives at the owners discretion). We could just enforce it for the main archive, but I would prefer to push people to do it the correct way, and save on surprises where a testbuild in to your PPA works, but then the real build in to the Ubuntu archive fails.

Implementation

bzr-builder will learn to read the same information as bzr-builddeb.

• If the package is marked as native then it will continue as before.
• If not then it will search for the pristine-tar data.
• If found it will create the tarball before building the package.
• If not then it will error if an option is set, or do nothing if not, building a native package as before.

In addition, Launchpad will have a way to decide if the last option should be set or not on builds. My vote would be for a "Daily Builds" boolean on IArchive. This will then be passed to slave to become a --option to the bzr-builder call.

Open Questions

What should the default for bzr-builder be?

How should LP know whether to use the option?

• Robert Collins: maybe bzr-builder should start depending on bzr-builddeb instead; having distro folk use a different tool to launchpad is a situation liable to lead to confusion and inhibit skill sharing. If bzr-builder were strictly layered on bzr-builddeb then it might be both easier to explain to distro folk and we'd have less 'change this twice' occurring. Your rationale about the design makes sense to me, though I'd like to raise one question - perhaps when the upstream version is changing daily, we should map that to a single upstream version and a debian version changing daily; few upstreams actaully /release/ daily, for all that they may /make tarballs/ daily.
  • I have explained before that we can't currently have bzr-builder use bzr-builddeb to build the packages, and that is still true.
  • You are wrong to assume that bzr-builder wouldn't depend on bzr-builddeb to share the code though.
  • It is up to the recipe writer whether they choose to change the upstream version daily, and I took both in to account in the text above. We can certainly put a recommendation in the documentation. Custom is for changing the upstream version when doing VCS snapshots, of which this is an extreme case. That's for generating tarballs though, and as we aren't going to do that changing the recommendation makes sense.