Build from branch into the primary archive

Short description of feature

Contact: MartinPool
LEP: https://dev.launchpad.net/LEP/BuildFromBranchIntoPrimary
On Launchpad: build-from-branch bugs

As an Ubuntu developer
I want Launchpad to build source and binary packages from branches
so that I don't have to both push the branch and also dput a source package

Rationale

At the moment, after making a change, Ubuntu developers must use bzr-builddeb to build a source package, and then dput to upload it. This is unnecessary work. Some developers only upload and don't push the branch.

By allowing Ubuntu developers to build directly from a branch into the primary archive of Ubuntu, we would:

• remove one unnecessary step.
• move closer to keeping all code in branches and being able to work with it there, thereby avoiding confusion about whether a branch is in use for a particular package or not.
• encourage use of branch-based release management features like merge proposals (currently hard to enforce since people can bypass them by uploading directly)

Risks

If the server-side building of packages does not work reliably, users will be blocked from uploading. We can allow a fallback to uploading some

Errors in packaging may be more difficult to see, understand, or debug if they happen on a server rather than locally. However, developers will still have the option to build the package locally.

Background:

• https://wiki.ubuntu.com/NoMoreSourcePackages

• https://wiki.ubuntu.com/DistributedDevelopment/

Stakeholders

• Ubuntu developers
• Linaro developers
• Ubuntu Technical Board
• James Westby
• Kiko Reis
• Ubuntu technical architect (Allison Randall) - read this 11 Feb
• Lamont Jones (interested in builder security)

Constraints and Requirements

Must

• Provide a way to build from current official package branches without uploading a source package and without using manually maintained recipes
• Expose this mechanism on the web UI
• Expose this mechanism via the web service

Since the aim is to provide a superior alternative to something that already exists within Launchpad, we must be at least as good as the current system. Specifically:

• As secure. There is great potential for damage here.
• As easy to track progress of a build
• As fast to build, at least
• As easy to use
• Be sufficiently secure
  • Systems are run under least-privileged and separation of powers, by Canonical IS
  • Perhaps: commits must be signed, so we have this check independent of them being sent up with a valid SSH key

Nice to have

• A mechanism for disabling dput uploads

  • Note that if we do the disabling, we'll definitely need to fix http://pad.lv/386596 (cannot push to a branch for a new source package)

  • Question: who should be able to control this?

Must not

Out of scope

Workflows

Publish a change into Ubuntu

1. User branches from the source package branch.
2. Make changes
3. Commit
4. Push
5. Request publication
6. How do we distinguish commits that ought to be built from those that don't? One way is to say we'll rebuild on things that add a new debian changelog (with a higher version.)

Success

How will we know when we are done?

• You can build from a branch directly into the primary Ubuntu archive

How will we measure how well we have done?

• Number of packages published in this way.

  • How are you going to measure this? -- jml <<Date(2011-01-31T13:35:23Z)>>

• Reported satisfaction

  • How are you going to measure this? -- jml <<Date(2011-01-31T13:35:23Z)>>

• Number of packages willing to proceed to turning off source package uploads

Thoughts?

• For integration testing, it seems like we need a place where Launchpad developers can publish changes into something that looks a lot like the primary Ubuntu archive, without actually being that archive. This could be a separate reserved pocket, or it could be a different Launchpad instance
• Whereas recipe builds use bzr-builder, this will probably use bzr-builddeb because no recipe is needed
  • We can have bzr-builder degrade to do the same thing as bzr-builddeb.
• Should packages build and publish on every commit, or should there be a separate request?
• Eventually it may be useful to have a per-package, per-distrorelease option to turn off source package uploads for areas we have decided branches are the best option. That isn't needed in the first iteration.
• The things missing from bzr-builder would be:
  • Control over the changelog message used
  • Better control over the resulting version number
  • GPG signatures required for something to end up in Ubuntu (API request isn't GPG signed)
  • Ability to produce non-native tarballs
• For the sake of testing, allow building from a branch into a ppa in a way that's just the same as what would happen on the distro.
• Speed/responsiveness of the remote server can be a hindrance to the workflow. Fairer scheduling of builds could help?
• Can still build locally for testing. Clarify how this would happen.
• Transition to IS-managed importer service, and others?
• Lack of staging areas? eg doing an upload to the toolchain, then rebuilding a large number of packages that depend on that thing. Can eventually automate creating a new ppa, building into that, looking at the results, then re-committing that into the primary archive branches.
• Concrete change: a command that asks lp over the api to please kick off a build of a branch, and do that rather than dput.
• Who will sign off that the security implications of this are ok? What are the security implications?
• Should we do this on top of recipes? Should they actually be exposed and editable or just behind the scenes?
  • Using recipes as they currently exist may make the linkage between source package branch and built source package much looser.
  • It seems like to locking the source package branch to the branch makes it easier to find the right source for a package, and easier to get the right access control.
• Expand on the story about committing changes that aren't yet ready to be built.

• How is this workflow discovered? Is this for official package branches only? How does this work for unofficial branches? What are the workflows for failure cases? -- jml <<Date(2011-01-31T13:35:23Z)>>

  • Mail to ubuntu-devel: we have the advantage here that everyone who will use this feature very likely reads that list
  • When a source package is assembled this way, we can make it obvious that it was built from a branch so that other people interested in it see it happening.
  • Expose it in the web ui? Where?
  • blog.launchpad.net
  • Soyuz upload-accepted mails could have a link to a fairly short web page.
  • Put this into the developer docs.
• We could try this out by running a bot on particular branches outside of Launchpad. That might be faster to develop. What would it prove?
  • Could have it write into a PPA rather than the primary archive? That avoids some security issues. But it won't actually be seen by developers. Does that really do much more than just setting up daily builds for those packages?

Changes needed

• Installer bzr-builder into buildd vms. (May already be done.)
• If we're not going to build on every commit, then there must be a mechansm to request a build. (Probably a web service api, and a bzr command that calls it.)
• New type of schedule-able job to request a source package be built from a branch. There is already a db enum for it, but we need to add a new build farm job type.

• The uploader (or something else?) should sign packages built by the buildds (with which key?). (bug 579870)

• After that, the binary can be built and published by the existing mechanism, so probably no other changes are needed.
• As a follow-on, add a mechanism to turn off/refuse dput uploads.