Branch Merge Queues

Branch merge queues are a way for us to continue a branch merge proposal through code review and follow it all the way into being merged into the branch, whether by a human, or a robot like Tarmac. Basically, it's PQM with Launchpad integration and without legacy arch support.

As a developer using Launchpad for development,
I want to queue merges to be handled by a bot
so that I can do more interesting work like actually writing code.

• Might be good to have a few more specific ones here. -- jml

Rationale

We are doing this now to reduce the pain the Launchpad team experiences in the current PQM/buildbot story. It will also provide benefits to existing users of tarmac (U1, OpenStack) by clarifying the automated merge story on Launchpad itself.

Stakeholders

• U1 - Ubuntu One uses Tarmac because it's much easier to set up than PQM, and much easier to maintain. They are 90% there, and would like some merge queues to make sure the story is complete. I talk with dobey, who is on the ops+ team on U1, regularly about this.
• Launchpad - Paul spoke with Gary at the Launchpad Epic about what the current issues are with the PQM/buildbot story. There are commonly so many moving parts that it makes it difficult to work with.

Constraints and Requirements

Must

• Support landing to many different branches on a single project (e.g. devel & db-devel with the Launchpad project)

• Support workflows where a merge must pass some automated process (e.g. a test suite). Note that this process cannot run within the Launchpad data center.
• Allow anyone with write permissions to the branch to move queue merges up and down in the queue
• Allow anyone with permission to queue a branch to unqueue it (essentially deleting it from the queue)
• Notify branch owner, merge proposal submitter and merge queue submitter when item is merged / fails to merge
• Queue must be discoverable by and visible to non-owners
• Contributors to a project must be able to quickly gain an understanding of how their patches will be merged
• Clearly indicate when merge is done and whether or not it is successful

Nice to have

• Where merges take a long time (e.g. Launchpad), some kind of visual progress indicator
• Mail interface
• Command-line client (perhaps Bazaar plugin)
  • Simple way to set up own machines for doing compilation / tests and reporting back

Must not

• Require any explicit configuration before use in projects which simply merge branches without testing/building
• Display details of security branches landing on public branches, in order to avoid leaking information about security vulnerabilities
• Require refreshing of the queue page to see updates to the queue

  • I'm not sure how our infrastructure is going to look when we get this far. We don't yet have a way of notifying the client with UI changes, and polling has proved to cause unneeded load on the appservers. This may be out of scope. -- rockstar

    • We don't yet have the infrastructure. We need to make it. We can't just keep adding refresh-driven crap. -- jml

Out-of-scope

• Actually running builds / tests for users within the Launchpad data centre

• Any system for storing structured test result data (see LEP/ResultsTracker)

• Providing web interface for inspecting historical queue data
• Allow one-button merging of a branch from a merge proposal page
• One-button merging of a branch that does not have a merge proposal
• Time estimates for queued merges
• APIs for gathering statistics about landings, e.g. (Tarmac can and will provide a plugin for this information)
  • Time
  • How often they fail

Workflows

• Add to queue
• Configure queue
• Merge succeeds in queue
• Merge fails in queue
• Re-order queue
• Freeze queue
• Unfreeze queue

See Code/MergeQueues for user stories and planning.

Success

How will we know when we are done?

We know we are done when U1 can completely abandon PQM for Tarmac and Merge Queues. As a side, we hope this also means that we've also satisfied all of the issues Launchpad-foundations needs, although their needs may be so specific that we can't put it squarely in Launchpad, and will have to build some external scripts (probably Tarmac plugins).

• Alternatively, perhaps we should simplify our needs until they can fit within Launchpad. -- jml

  • Maybe. I basically will know I'm done based on the use case of U1 itself. I feel like they have pretty good handle on their needs. -- rockstar

How will we measure how well we have done?

Paul would like to set up a few graphs to see this:

• Total number of merge queues created
• Number of merge queue items created in a single day
• Number of merge queue items completed in a single day
• Number of branches merged by one queue

jml would like to see:

• Number of projects using merge queues compared to number of projects that actually merge branches
• For projects that use merge queues, how many merges done with queue vs how many done manually
• How many projects outside Canonical actually use merge queues

Thoughts?

• The vast majority of cases are going to be focused on merging to trunk. Let's make sure the UI reflects this.
  • Should we only allow series branches to be managed by a queue?
    • Don't know. Maybe make it hard to do anything else, but not quite forbid doing anything else? -- jml
• Although we have to support many branches to handle Launchpad's case, perhaps we can simplify the UI by only allowing one queue per project.
  • Paul doesn't agree with this, since a user might want to maintain a fork, and yet still have it managed by a queue. Also, basing them on projects means that source package branches can't use them, and that would be leaving out the whole UDD story. That's a pretty significant thing to be leaving out.

    • Good point re package branches. Regarding the fork, we do want to support that use case, but we need to do it in such a way that it doesn't create confusion or extra UI burden for the vast majority of non-forked projects. -- jml

• For a test running bot, one strategy is to optimistically merge everything, run tests, and then bisect in case of failure. It would be nice if this strategy were supported by our web UIs. jml doesn't know how this would look, though.
  • Yeah, so this is something for the Tarmac side probably, but Gary and Paul have talked about this. Paul doesn't think it needs to be dealt with on the Launchpad side, but on the merge bot side.
    • Sounds fair enough. -- jml