launchpad development|
Size: 4582
Comment: Link to PQM.
|
← Revision 18 as of 2019-10-04 11:26:34 ⇥
Size: 3248
Comment: update for git
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 7: | Line 7: |
| * [[https://launchpad.net/pqm|PQM]] -- this is how changes get into the devel and db-devel branches | * [[https://jenkins.ols.canonical.com/online-services/job/launchpad/|Jenkins]] -- this is how changes get into the master and db-devel branches |
| Line 11: | Line 11: |
| There is a (currently private) [[https://code.edge.launchpad.net/~launchpad-pqm/lpbuildbot/trunk|branch on launchpad]] that contains the buildbot config and the buildbot-poll.py script. | There is a (currently private) [[https://code.launchpad.net/~canonical-launchpad-branches/lpbuildbot/production|branch on launchpad]] that contains the buildbot config and the buildbot-poll.py script. |
| Line 15: | Line 15: |
| == PQM == | == Jenkins == |
| Line 17: | Line 17: |
| [[https://launchpad.net/pqm|PQM]] is a robot that processes requests to merge branches into either devel or db-devel. | We use the [[https://jenkins.ols.canonical.com/online-services/|Canonical Online Services team's Jenkins instance]] to process requests to merge branches into either master or db-devel. It is currently private. |
| Line 19: | Line 19: |
| For devel, all the requests come from developers. For db-devel, many requests are generated by buildbot-poll.py merging stable into db-devel. PQM checks that the commit message matches the required regexp, that the branch merges cleanly into the target branch, and runs "make check_merge" for landings to devel and "make check_db_merge" for landings to db-devel. The only difference is that "make check_merge" tries to prevent database changes landing. PQM has two configs: the default config and the 'testfix' config. The difference is that the 'testfix' config requires the commit message start with '[testfix]'. Sadly, there is no way of telling which config PQM is using at present, other than submitting merge requests (see [[https://bugs.edge.launchpad.net/launchpad-foundations/+bug/424060|bug 422964]]). The buildbot-poll.py script is responsible for switching between the configs. |
At the moment, Jenkins merges any merge proposal that has an Approve vote from at least one Launchpad reviewer, whose status is Approved ("top-approved"), and that has no conflicts against its target branch. It does not run any tests itself, leaving that to buildbot. |
| Line 33: | Line 25: |
| At one level, it's fairly simple: when a change is detected on the devel or db-devel branches on launchpad, the tests are run on that branch and their success or failure noted. Buildbot also runs some non-trunk builds -- for example, testing devel against bzr.dev -- but these are ignored for the purposes of this page. |
At one level, it's fairly simple: when a change is detected on the master or db-devel branches on launchpad, the tests are run on that branch and their success or failure noted. |
| Line 39: | Line 29: |
| The change source we use is the "Bzr``Poller" in bzrbuildbot/poller.py in the lpbuildbot branch. It is configured with a list of URLs to watch and when it sees a new revision in one of these branches, it feeds it to buildbot. | The change source we use is the customised version of "Git``Poller" in bzrbuildbot/poller.py in the lpbuildbot branch. It is configured with a list of URLs to watch and when it sees a new revision in one of these branches, it feeds it to buildbot. |
| Line 41: | Line 31: |
| The scheduler we use for the two trunk builders is "Aggregating``Testfix". An Aggregating``Testfix scheduler is configured with a branch and watches for changes that affect this branch. When it sees a change that affect its branch, it checks to see if the last build succeeded or failed. If it failed, then it only starts a new build if the commit message contains '[testfix]'. | The scheduler we use for the two trunk builders is "Aggregating``Scheduler". An Aggregating``Scheduler is configured with a branch and watches for changes that affect this branch. When it sees a change that affect its branch, it checks to see if the last build succeeded or failed. If it failed, then it only starts a new build if the commit message contains '[testfix]'. |
| Line 43: | Line 33: |
| We have a custom web UI that adds: | We have a custom web UI that adds a page that lets you force a build. (Buildbot's built-in "Force Build" button doesn't work for us for rather boring reasons.) |
| Line 45: | Line 35: |
| * a page that lets you force a build, and * an XML-RPC method used by buildbot-poll.py. (Builbot's built-in "Force Build" button doesn't work for us for rather boring reasons). Builds forced using our page have a distinctive "reason" attribute that the buildbot-poll.py script looks for using our custom XML-RPC method. |
Builds forced using our page have a distinctive "reason" attribute that the buildbot-poll.py script looks for using the JSON API. |
| Line 54: | Line 39: |
| This script checks the status of the builds (via XML-RPC) on buildbot and does two main things depending on what it finds: | This script checks the status of the builds (via the JSON API) on buildbot, and may push to the stable and/or db-stable branches depending on what it finds. |
| Line 56: | Line 41: |
| 1. it swaps PQM in and out of 'testfix' mode. 1. it may push to the stable and/or db-stable branches. |
For each development branch (i.e. master or db-devel), if the most recent build succeeded the script pushes the revision of the development branch that was tested into the corresponding stable branch (i.e. stable or db-stable). |
| Line 59: | Line 43: |
| PQM is put in testfix mode if either branch is unclean, where "unclean" means "the last test run failed and we haven't had a testfix revision or a forced build yet". We check the branch itself for the testfix revision, so buildbot doesn't have to have seen it before we go out of testfix. If a build is forced in the UI, buildbot-poll.py can't tell until the build has started and this doesn't happen until the EC2 instance has booted which can take a few minutes. For each development branch (i.e. devel or db-devel), if the most recent build succeeded the script pushes the revision of the development branch that was tested into the corresponding stable branch (i.e. stable or db-stable). It runs out of cron every 5 minutes, on the PQM box (it has to run on the PQM box, because it switches in and out of testfix mode by moving config files around). |
It runs out of cron every 5 minutes, on the PQM box. |
How the 4 trunk branches and buildbot work together
This page aims to explain the mechanisms that make the wonderful diagrams on the "where is trunk?" page work. You'll need to read that page for this one to make any sense at all.
There are three main components:
Jenkins -- this is how changes get into the master and db-devel branches
buildbot -- this is what runs the tests.
- buildbot-poll.py -- this script monitors the branches and builds on buildbot and implements much of our policy.
There is a (currently private) branch on launchpad that contains the buildbot config and the buildbot-poll.py script.
The buildbot UI is currently also private, but this will change, hopefully soon.
Jenkins
We use the Canonical Online Services team's Jenkins instance to process requests to merge branches into either master or db-devel. It is currently private.
At the moment, Jenkins merges any merge proposal that has an Approve vote from at least one Launchpad reviewer, whose status is Approved ("top-approved"), and that has no conflicts against its target branch. It does not run any tests itself, leaving that to buildbot.
Buildbot
Buildbot is our continuous integration tool of choice.
At one level, it's fairly simple: when a change is detected on the master or db-devel branches on launchpad, the tests are run on that branch and their success or failure noted.
In general in buildbot, a "change source" produces "changes" which are fed to a "scheduler" which can examines the change to determine whether to trigger a build.
The change source we use is the customised version of "GitPoller" in bzrbuildbot/poller.py in the lpbuildbot branch. It is configured with a list of URLs to watch and when it sees a new revision in one of these branches, it feeds it to buildbot.
The scheduler we use for the two trunk builders is "AggregatingScheduler". An AggregatingScheduler is configured with a branch and watches for changes that affect this branch. When it sees a change that affect its branch, it checks to see if the last build succeeded or failed. If it failed, then it only starts a new build if the commit message contains '[testfix]'.
We have a custom web UI that adds a page that lets you force a build. (Buildbot's built-in "Force Build" button doesn't work for us for rather boring reasons.)
Builds forced using our page have a distinctive "reason" attribute that the buildbot-poll.py script looks for using the JSON API.
buildbot-poll.py
This script checks the status of the builds (via the JSON API) on buildbot, and may push to the stable and/or db-stable branches depending on what it finds.
For each development branch (i.e. master or db-devel), if the most recent build succeeded the script pushes the revision of the development branch that was tested into the corresponding stable branch (i.e. stable or db-stable).
It runs out of cron every 5 minutes, on the PQM box.
