Build from branch into the primary archive
Short description of feature
On Launchpad: bfbip bugs
As an Ubuntu developer
I want Launchpad to build source and binary packages from branches
so that I don't have to both push the branch and also dput a source package
At the moment, after making a change, Ubuntu developers must use bzr-builddeb to build a source package, and then dput to upload it. This is unnecessary work. Some developers only upload and don't push the branch.
By allowing Ubuntu developers to build directly from a branch into the primary archive of Ubuntu, we would:
- remove one unnecessary step.
- move closer to keeping all code in branches and being able to work with it there, thereby avoiding confusion about whether a branch is in use for a particular package or not.
- encourage use of branch-based release management features like merge proposals (currently hard to enforce since people can bypass them by uploading directly)
If the server-side building of packages does not work reliably, users will be blocked from uploading. We can allow a fallback to uploading some
Errors in packaging may be more difficult to see, understand, or debug if they happen on a server rather than locally. However, developers will still have the option to build the package locally.
Since there will be automatic systems that both import packages to branches, and that build packages from branches, there is some risk of a positive-feedback failure. This could be guarded against by, for instance, blacklisting any auto-imported revisions from being rebuilt into a package.
- Ubuntu developers
- Linaro developers
- Ubuntu Technical Board
- James Westby
- Kiko Reis
- Ubuntu technical architect (Allison Randall) - read this 11 Feb
- Lamont Jones (interested in builder security)
Constraints and Requirements
- Provide a way to build from current official package branches without uploading a source package and without using manually maintained recipes
- Expose this mechanism on the web UI
- Expose this mechanism via the web service
Since the aim is to provide a superior alternative to something that already exists within Launchpad, we must be at least as good as the current system. Specifically:
- As secure. There is great potential for damage here.
- Systems are run under least-privileged and separation of powers, by Canonical IS
- Commits must be signed by the developer, in addition to being pushed over SSH
- There must be an audit trail from signature through to build
Past experience shows that it's worth being crystal clear on what sort of reports you want from this audit trail
- As easy to track progress of a build
- As fast to build, at least
- As easy to use
- Must not be slower than the current system
- High demand for binary builds should not excessively delay builds of source packages (might need a separate queue)
Nice to have
Out of scope
- Disabling source package uploads
Publish a change into Ubuntu
- User makes a local branch from the source package branch.
- Make changes
- Push back to source package branch
- Request publication
- How do we distinguish commits that ought to be built from those that don't? One way is to say we'll rebuild on things that add a new debian changelog (with a higher version.) Some people commit changes with a series target of 'unreleased' and we could then just actually assemble the package when that flips to be a real series.
How will we know when we are done?
- You can build from a branch directly into the primary Ubuntu archive
How will we measure how well we have done?
- Number of packages published in this way.
How are you going to measure this? -- jml <<Date(2011-01-31T13:35:23Z)>>
- Reported satisfaction
How are you going to measure this? -- jml <<Date(2011-01-31T13:35:23Z)>>
- For integration testing, it seems like we need a place where Launchpad developers can publish changes into something that looks a lot like the primary Ubuntu archive, without actually being that archive. This could be a separate reserved pocket, or it could be a different Launchpad instance
- Should packages build and publish on every commit, or should there be a separate request?
- Eventually it may be useful to have a per-package, per-distrorelease option to turn off source package uploads for areas we have decided branches are the best option. That isn't needed in the first iteration.
- The things missing from bzr-builder would be:
- GPG signatures required for something to end up in Ubuntu (API request isn't GPG signed)
- For the sake of testing, allow building from a branch into a ppa in a way that's just the same as what would happen on the distro.
- Speed/responsiveness of the remote server can be a hindrance to the workflow. Fairer scheduling of builds could help?
- Can still build locally for testing. Clarify how this would happen.
- Transition to IS-managed importer service, and others?
- Lack of staging areas? eg doing an upload to the toolchain, then rebuilding a large number of packages that depend on that thing. Can eventually automate creating a new ppa, building into that, looking at the results, then re-committing that into the primary archive branches.
- Concrete change: a command that asks lp over the api to please kick off a build of a branch, and do that rather than dput.
- Who will sign off that the security implications of this are ok? What are the security implications?
- Should we do this on top of recipes? Should they actually be exposed and editable or just behind the scenes?
- Using recipes as they currently exist may make the linkage between source package branch and built source package much looser.
- It seems like to locking the source package branch to the branch makes it easier to find the right source for a package, and easier to get the right access control.
- wgrant agrees with doing it on top of recipes but not exposing them.
- Expand on the story about committing changes that aren't yet ready to be built.
How is this workflow discovered? Is this for official package branches only? How does this work for unofficial branches? What are the workflows for failure cases? -- jml <<Date(2011-01-31T13:35:23Z)>>
- Mail to ubuntu-devel: we have the advantage here that everyone who will use this feature very likely reads that list
- When a source package is assembled this way, we can make it obvious that it was built from a branch so that other people interested in it see it happening.
- Expose it in the web ui? Where?
- Soyuz upload-accepted mails could have a link to a fairly short web page.
- Put this into the developer docs.
- We could try this out by running a bot on particular branches outside of Launchpad. That might be faster to develop. What would it prove?
- Could have it write into a PPA rather than the primary archive? That avoids some security issues. But it won't actually be seen by developers. Does that really do much more than just setting up daily builds for those packages?
- wgrant doesn't see much value in doing an external prototype.
- We could include the testament signature or hash within the source package metadata to give some traceability back to what was signed by a person.
- Probably the bot that assembles the source packages will sign them with a GPG key for integrity, but this won't be a semantic signature in the same way that a signature by a developer could be.
- Do this first mostly with Linaro.
- How is this going to affect syncs from Debian? We could keep the current sync-source mechanism active, also for translations uploads, dist-upgrade uploads.
The Derived Distros feature is intended to handle Debian syncs, but no thought has been put into making it work with package branches. Bear that in mind! -- Julian 2011-03-24
The current mail notifications used for recipe uploads to PPAs will annoy people if used for Ubuntu because it sends email for recipe build successes. I suggest that the notification special-casing for recipes is removed entirely and we just let the existing notifications do the job - correctly. -- Julian 2011-03-24
- A new bzr-builder needs to be deployed on Launchpad, which adds support for .orig tarballs and building directly from a branch without a recipe (RT 46345)
- If we're not going to build on every commit, then there must be a mechanism to request a build. (Probably a web service api, and a bzr command that calls it.)
- New type of schedule-able job to request a source package be built from a branch. There is already a db enum for it, but we need to add a new build farm job type.
The uploader (or something else?) should sign packages built by the buildds (with which key?). (bug 579870)
- After that, the binary can be built and published by the existing mechanism, so probably no other changes are needed.
bzr-builder should include a testament or manifest among the files it uploads (bug 747237)