Not logged in - Log In / Register


Overall Summary

Summary: Generalise the soyuz build system so that any job can be sent to the build farm, not just build jobs. Create a new BuilderRequest table to link the BuildQueue to specific job types, e.g. a new table LiveImageBuild that will contain LiveCD ISOs, or BranchBuild that will contain Source Package Branch builds

Goal/Deliverables: The initial deliverable will let us build source package branches in Launchpad.

We will know we have finished when we can build arbitrary jobs on the existing build farm, e.g. source package branches.

Release Note


Use cases


User Interface

Not strictly related to buildd-generalisation, but for source package branch building:

High-level implementation points

If we write the system to run generic jobs (should we?) then "builder" seems like the wrong term. -- jml
I would much rather keep the existing nomenclature for builders, at least for now, to reduce confusion and keep my sanity -- Julian

Implementation Plan


Maybe a good early step here is to take the existing implementation and refactor it so that it's in terms of Jobs? -- jml
I think this is a lot of work with little benefit. If you want to use an IJob interface to it, it would be better to add that as a layer on top of the existing implementation so it pushes jobs at the build farm. -- Julian
And in fact this is exactly what I'm going to do now. -- Julian

Code Changes


Remove build column.
Remove these properties, and make call sites cope:

Delegate score() through IXXXJob classes so job-specific criteria is used to score.



IBuilder refactoring

New model classes

Schema Changes

--- Variant One ---

We need to generalise builder jobs. In order to model different types of builder requests (while avoiding nullable foreign keys in the BuildQueue table) This schema is proposed:

Amend BuildQueue to add one new column:

builderrequest (FK)


In light of the fact that the BuildQueue table now contains rows which are not necessarily build jobs, it should be given a more suitable name like JobQueue or BuilderQueue.

Amend Build so that its 'id' column becomes a foreign key to the BuilderRequest parent table:

id (int) (FK to BuilderRequest (on DELETE CASCADE))


New table BuilderRequest:

id (int)

request_type (text)

requested_by (text)

date_requested (date)

duration (interval)

date_finished (date)

New table LiveImageBuild:

id (int) (FK to BuilderRequest (on DELETE CASCADE))

archive (FK)

distroarchseries (FK)

description (text)

image_file (FK to LFA)

status (int)

Index: btree(archive, distroarchseries)

The BuilderRequest table is a parent table. It thus delegates its key values to the child tables that contain data pertinent to specific builder requests. Please note that the deletion of a parent table row should result in the deletion of the corresponding child table row as well.

Jobs for builders are queued in the BuildQueue table as they currently are, except it will not have a FK directly pointing to a Build. Instead it has a FK to a BuilderRequest row. That row contains an ID and a type. The type tells which child table to join to, and the child table contains a foreign key pointing back to the BuilderRequest.



--- Variant two ---

Similar to Variant One, but:

  1. the JobQueue table takes on the role of the parent table.

  2. the BuilderRequest table is not needed.

  3. the (Build|LiveImageBuild) tables share 'id' values with the JobQueue table without a foreign key constraint since the JobQueue rows will be deleted once a job completes.

The 'build' FK is removed from the table Build/Jobqueue and a 'job_type' column is added:


job_type (text)


The Build and LiveImageBuild tables are amended to contain the following additional columns:


requested_by (text)

date_requested (date)

duration (interval)

date_finished (date)




--- Variant three ---

We don't use any parent/child table techniques but add a nullable foreign key for each job type to the JobQueue table.



--- Variant four --- Another parent/child relationship, but this one lets us keep the current meme that BuildQueue rows are temporary.

  1. New table BuildJob that represents the common job data for all jobs and has an enum "type" column for the type of job

  2. BuildQueue gains a new FK column to BuildJob

  3. BuildJob has child tables that represent the job-specific data, such as BuildJobBranch and BuildJobCDImage

  4. The child tables' IDs are the same as the relevant parent BuildJob table's and are in fact a FK to BuildJob.






Unresolved issues

CategoryProposal CategoryTemplate

Soyuz/Specs/BuilddGeneralisation (last edited 2009-12-18 10:21:28 by michael.nelson)