launchpad development
How the 4 trunk branches and buildbot work together
This page aims to explain the mechanisms that make the wonderful diagrams on the "where is trunk?" page work. You'll need to read that page for this one to make any sense at all.
There are three main components:
- PQM -- this is how changes get into the devel and db-devel branches
- buildbot -- this is what runs the tests.
- buildbot-poll.py -- this script monitors the branches and builds on buildbot and implements much of our policy.
There is a (currently private) branch on launchpad that contains the buildbot config and the buildbot-poll.py script.
The buildbot UI is currently also private, but this will change, hopefully soon.
PQM
PQM is a robot that processes requests to merge branches into either devel or db-devel.
For devel, all the requests come from developers.
For db-devel, many requests are generated by buildbot-poll.py merging stable into db-devel.
PQM checks that the commit message matches the required regexp, that the branch merges cleanly into the target branch, and runs "make check_merge" for landings to devel and "make check_db_merge" for landings to db-devel. The only difference is that "make check_merge" tries to prevent database changes landing.
PQM has two configs: the default config and the 'testfix' config. The difference is that the 'testfix' config requires the commit message start with '[testfix]'. Sadly, there is no way of telling which config PQM is using at present, other than submitting merge requests.
The buildbot-poll.py script is responsible for switching between the configs.
Buildbot
Buildbot is our continuous integration tool of choice.
At one level, it's fairly simple: when a change is detected on the devel or db-devel branches on launchpad, the tests are run on that branch and their success or failure noted.
Buildbot also runs some non-trunk builds -- for example, testing devel against bzr.dev -- but these are ignored for the purposes of this page.
In general in buildbot, a "change source" produces "changes" which are fed to a "scheduler" which can examines the change to determine whether to trigger a build.
The change source we use is the "BzrPoller" in bzrbuildbot/poller.py in the lpbuildbot branch. It is configured with a list of URLs to watch and when it sees a new revision in one of these branches, it feeds it to buildbot.
The scheduler we use for the two trunk builders is "AggregatingTestfix". An AggregatingTestfix scheduler is configured with a branch and watches for changes that affect this branch. When it sees a change that affect its branch, it checks to see if the last build succeeded or failed. If it failed, then it only starts a new build if the commit message contains '[testfix]'.
buildbot-poll.py
This script checks the status of the builds (via XML-RPC) on buildbot and does two main things depending on what it finds:
- it swaps PQM in and out of 'testfix' mode.
- it may push to the stable and/or db-stable branches.
PQM is put in testfix mode if either branch is unclean, where "unclean" means "the last test run failed and we haven't had a testfix revision yet". We check the branch itself for the testfix revision, so buildbot doesn't have to have seen it before we go out of testfix.
For each development branch (i.e. devel or db-devel), if the most recent build succeeded the script pushes the revision of the development branch that was tested into the corresponding stable branch (i.e. stable or db-stable).
It runs out of cron every 5 minutes, on the PQM box (it has to run on the PQM box, because it switches in and out of testfix mode by moving config files around).
