Build Farm Builder Pools

Make pools of builders reserved for usage by different groups of users.

As a user of daily builds
I want to make sure the builds are done without undue delay
so that so that the latest builds are available promptly

As an Ubuntu developer
I want to make sure that daily builds don't swamp the build farm
so that regular Ubuntu package builds are not delayed

• Groups of builders may be owned by different teams, e.g. OEM could own a set of armel builders that are exclusively for OEM builds
• Builders may also have an affinity to certain build types
  • Non-exclusivity allows other builds to use those builders when they're idle
  • Build jobs with affinity to those builders will take priority

Rationale

Why are we doing this now?

1. The daily builds project will be coming on line soon and extra hardware will be purchased for its use. We want to make sure that hardware really does get used for daily builds.
2. The OEM team wants a pool of armel builds for its own use

What value does this give our users? Which users?

• This feature ensures that daily builds won't monopolise the build farm over everything else.
• Conversely, it ensures that everything else won't monopolise the build farm to the detriment of daily builds.
• It helps OEM to do timely builds using their own buildd hardware.

Stakeholders

Who really cares about this feature? When did you last talk to them?

• Mark S.
• OEM

Constraints

• At the very least, a set of builders exclusively for daily builds.
• Daily builds must use the daily build pool of builders
• OEM must be able to use PPAs to get timely builds of their packages
• Web-based admin control of build pools
  • Admins must be able to assign builders to a pool
  • Reassign builders from one pool to another
  • Remove builders from a pool
  • Create new pools
  • Rename pools
  • Delete pools
  • Decide which pool of builders should be used for a given archive, if any
• When a pool is reserved for a particular group, e.g. OEM, it must be possible for members of that group to edit their archives to build using that pool. It must not be possible for anyone else except admins to do this.

This feature does not provide:

• A collected queue of builds scheduled for a specific pool
• Any in-Launchpad mechanism to measure trends in build-times

Workflows

Review existing pools

If a buildd-admin wants to see a list of all the existing pools, they visit /builders/+pools and get a sortable table showing the pools along with the PPAs that are using them. The admin can then click on a pool name to edit it.

Note: "exclusive" means that only the archives associated with those pools may use them. If a pool is not exclusive, it means that any other archive can build on it, but at a lower priority such that associated archives will always jump the queue with new builds.

Create a new pool

From the list of pools page, a buildd-admin can create a new pool by visiting /builders/+pools/+new.

Once the pool is created, the page redirects to /builders/+pools/<name>.

Edit an existing pool

A buildd-admin can visit this page to edit or view an existing pool. Editing an existing pool will happen if new builder hardware is bought and needs to be added to a pool. This page can be used to add or remove builders from a pool.

The builder:+edit page will also be changed allow a buildd-admin to change its pool via the standard form.

Set pool for archive

An OEM engineer or Buildd-admin visits the Archive:+edit page and sees an enabled widget that allows multi-selection of available (i.e. permissioned) pools.

Review builders

A buildd-admin visits /builders and clicks on the "pool" column to sort the builders by the pools that they're in.

Success

We will know we are done when daily builds will only build on their designated builders and don't cause disruption to the other build types on the build farm.

We can measure our success by checking that:

• Number of blocked builds goes down, and duration of blockages goes down.
• That there is a predictable, short wait time for building such that we can reasonably establish an SLA with certain build pool owners.

Thoughts?

jml doesn't know what these two mean any more:

• Archives need multiple pools so that they can access different sets of restricted builders (e.g. different architectures may be added)

  • jml: for the first one consider where we have a pool of armel builders and a pool of DailyBuild - we might conceivably want to add armel support to a daily build. --Julian

• We will restrict who can create daily builds to start with, hence the restrictions on who can give their PPAs access.

  • We don't want world+dog creating daily builds until we're comfortable with the infrastructure and build capacity. --Julian

Implementation notes

• Bjorn wonders if the exclusivity flag should be on the builder table or the builderpool table?
• We need to look at fixing determineArchitecturesToBuild() so that it takes pools into account.