Diff for "Soyuz/TechnicalDetails/Building"

Not logged in - Log In / Register

Differences between revisions 7 and 8
Revision 7 as of 2017-06-20 09:38:12
Size: 3162
Editor: cjwatson
Comment: modernise a bit
Revision 8 as of 2017-06-20 09:38:43
Size: 3132
Editor: cjwatson
Comment: remove dead source ref
Deletions are marked like this. Additions are marked like this.
Line 7: Line 7:
 * lib/lp/soyuz/model/build*

Next Page: Publishing

Soyuz Building

  • lib/lp/buildmaster/
  • lib/lp/soyuz/model/binarypackagebuild*

Launchpad has a build farm capable of building different types of builds, such as Bazaar source recipes and translations files, but here we will concentrate on the Soyuz aspects: turning a source package into a binary package (or packages).

There are two types of builder, virtual and non-virtual. The non-virtual builders are only used for a small number of architectures where we do not yet have adequate virtualisation infrastructure. All other builds are done on virtual builders.

Buildd Manager

The buildd-manager is a Twisted daemon that is responsible for dispatching builds to the builders, monitoring the builders (including disabling failed ones) and collecting files for finished builds. There is a table called BuildQueue that contains a record of all outstanding or in-progress builds. When the build completes, the relevant record is removed from the table. More information about the buildd-manager can be found at XXX.

More about build creation

When a source is accepted and a build created for it, several pre-requisite conditions need to be met:

  • The source's requested architectures must be buildable in the target DistroSeries, by virtue of a DistroArchSeries record being present with a matching architecture tag.

  • There must be a builder available that builds the requested architecture(s).
  • There must be a chroot file available for the DistroArchSeries. (see below for more about chroots)

Managing builders and chroots

Buildd admins manage the build farm. One of the important tasks is creating chroot images. These are a "tarball" of the initial filesystem layout required to start a build. When a build starts on a builder, it unpacks the tarball sent to it from the manager and "chroots" inside it. Each DistroArchSeries has its own chroot file, which is periodically updated so that the builders have a quicker start-up time so that apt-get dist-upgrade has less work to do at the start of each build.

The manage-chroot script in lp:ubuntu-archive-tools deals with adding and removing chroot tarballs in the librarian. It uses the setChroot and removeChroot methods on DistroArchSeries via the webservice.

Processing builds

<Add DB schema diagram here>

After the buildd-manager collects the built files from the builder, it puts them in an "incoming" queue area, very similar to where txpkgupload puts its source uploads. There is a separate cron job that runs process-upload.py on the buildmaster machine which works almost identically to processing sources, except it runs with an upload policy that allows unsigned uploads.

After the build upload is processed, the build record serves as a link between the source and its binary. This makes for some fun SQL queries all over Soyuz.

Next Page: Publishing

Soyuz/TechnicalDetails/Building (last edited 2017-06-20 09:38:43 by cjwatson)