launchpad development
Soyuz Building
- lib/lp/buildmaster/
- lib/lp/soyuz/model/binarypackagebuild*
- lib/lp/soyuz/model/build*
Launchpad has a build farm capable of building different types of builds, such as Bazaar source recipes and translations files, but here we will concentrate on the Soyuz aspects: turning a source package into a binary package (or packages).
There are two types of builder, virtual and non-virtual. The non-virtual builders are currently only used by Ubuntu and the Ubuntu-security team PPA. All other builds are done on virtual builders.
Buildd Manager
The buildd-manager is a Twisted daemon that is responsible for dispatching builds to the builders, monitoring the builders (including disabling failed ones) and collecting files for finished builds. There is a table called BuildQueue that contains a record of all outstanding or in-progress builds. When the build completes, the relevant record is removed from the table. More information about the buildd-manager can be found at XXX.
More about build creation
When a source is accepted and a build created for it, several pre-requisite conditions need to be met:
The source's requested architectures must be buildable in the target DistroSeries, by virtue of a DistroArchSeries record being present with a matching architecture tag.
- There must be a builder available that builds the requested architecture(s).
There must be a chroot file available for the DistroArchSeries. (see below for more about chroots)
In addition, if the build is for a distribution (as opposed to a PPA) then a special so-called P-A-S (Package-Arch-Specific) file is consulted to see whether to either force or disallow a build completely for certain package names on certain architectures. This is useful in situations for example where we know the package has a bug on a particular architecture and it needs to be blocked until the bug is fixed.
Managing builders and chroots
It's currently a function of the Buildd Admins (part of IS) to manage the build farm, but this function will move to the LOSAs in the near future. One of the important tasks is creating chroot images. These are a "tarball" of the initial filesystem layout required to start a build. When a build starts on a builder, it unpacks the tarball sent to it from the manager and "chroots" inside it. Each DistroArchSeries has its own chroot file, which is periodically updated so that the builders have a quicker start-up time so that apt-get dist-upgrade has less work to do at the start of each build.
There is a script scripts/ftpmaster-tools/manage-chroot.py that adds and removes chroot tarballs in the librarian. It currently has to be run inside the data centre because we have no way of dealing with ~640M tarballs remotely.
Processing builds
<Add DB schema diagram here>
After the buildd-manager collects the built files from the builder, it puts them in an "incoming" queue area, very similar to where Poppy puts its source uploads. There is a separate cron job that runs process-upload.py on the buildmaster machine which works almost identically to processing sources, except it runs with an upload policy that allows unsigned uploads.
After the build upload is processed, the build record serves as a link between the source and its binary. This makes for some fun SQL queries all over Soyuz.
