This is a proposal of how a developer branch flows through the different production branches we have, after the developer branch is ready for landing.

Merge Workflow Draft 2

Bugfixes land on the stable branch, and are rolled out daily to a "QA testing" server for QA. Revisions that have been marked QA-OK are copied to a production branch, and then rolled out daily to production and edge.

Branches that contain db changes land on the db-stable branch, and are QA'd on staging. DB changes are rolled out monthly, as today.

(Old image: merge-workflow-draft-2.jpeg)

The QA step

A revision gets merged from stable to production only after it has been marked as good in the QA step. This helps ensure that we don't rollout feature to launchpad.net that haven't been QAed.

In addition to this, a revision only gets merged from stable to production if all the previous revisions have been marked as good as well. This is because bzr doesn't support cherry picking, and we want to keep track of which revisions have been merged.

How the QA step works regarding to marking the revision as good/bad is described here: https://dev.launchpad.net/QAProcessContinuousRollouts.

What if I don't do my QA, or my QA is bad?

If someone does not do their QA, or their branch fails QA, then they will block production/edge rollouts until that revision is reverted. Rollouts from stable to the QA server will still continue, so other developers can still continue their QA as normal.

We assume a two day grace period for QA - if on the third day a branch has not been QA'd, then anyone on the team may revert the offending revision.

• We must focus on making these reversions as easy and painless as possible.

What if I have to do QA that takes a few days?

If a feature requires more than one day of QA, then developers have the option to do the QA on the staging.launchpad.net server, to requisition an additional server, (dogfood), or ask for permission to block the daily QA rollouts for more than two or three days.

The Rollout

Rollouts of database changes still happen as before.

Large Features That Span Branches

If you have a feature that will span multiple branches, then it is expected that the developers will create a feature switch to optionally enable that part of the code, or that early branches that have no user-facing changes will be marked as qa-untestable.

Emergency Fixes/Cherry Picks

The cherry-pick procedure remains the same as it does currently - you must land the change on both the production branch and later on the stable branch.

Merge Workflow Draft 1

To summarize, branches that contain db changes always get landed to the db branch. Bug fixes get landed to the stable branch, and larger features (consisting of multiple branches) get landed to an integration branch for that feature.

For bug fixes, the landed revision stays in the stable branch until it has been marked as being QAed. When the revision is marked as good, it automatically gets merged into the production branch, which is rolled out to launchpad.net daily.

For larger features, the integration branch gets combined with stable, and rolled out to edge.launchpad.net daily. When the feature is done and QAed, it gets merged into the stable branch, and the merged revision finally gets merged into the production branch after it has been QAed again on staging.

Image source: merge-workflow.dia

The QA step

A revision gets merged from stable to production only after it has been marked as good in the QA step. This helps ensure that we don't rollout feature to launchpad.net that haven't been QAed.

In addition to this, a revision only gets merged from stable to production if all the previous revisions have been marked as good as well. This is because bzr doesn't support cherry picking, and we want to keep track of which revisions have been merged.

XXX: How the QA step works regarding to marking the revision as good/bad is still undefined.

Feature branches

Features that will involve multiple branches should have an integration branch for that feature. Instead of landing branches related to the feature to stable, they land their branches to the integration branch. New revisions in the integration branch will automatically be merged into the edge branch.

The reason for having an integration branch branch is to keep track of all the revisions that belong to the feature, so that when the feature has been marked as good in the QA process, the whole feature can be rolled out at once. Parts of the feature can still be rolled out by merging specific revisions into stable.

XXX: How to register new feature branches, and how to merge to them, is still undefined. Maybe a better way would be to use revision properties to tag revisions as being part of a bigger feature, or use bug links?

Resolving conflicts

If there's a conflict when automatically merging the feature branch into the edge branch, someone has to manually merge the feature branch into edge and resolve the conflicts.

Bug fixes

Bug fixes for features that are already exposed on launchpad.net can go directly to the stable branch. After the revision has been QAed, it gets merged into the production branch.

When to use a feature branch

So far we've talked about features and bug fixes. However, all features don't need a feature branch. A feature branch is only necessary when you have a larger feature that will require multiple branches to complete. If you have a small feature, or improvement of an existing feature, that you're quite sure will require only one branch, landing it directly on the stable branch is fine.

Which branch should you branch off from when starting new work?

When starting new work, you should branch off from the db branch if your work will require db changes, and from the stable branch otherwise. You shouldn't start of from edge, since it contains features that shouldn't be rolled out to production yet, which means that you would have to wait for the existing features to be finished, before your feature can be considered for rollout.

If possible, the required db changes should go to the db branch one cyle in advance, to make it possible to QA the other work on edge.

Continuous rollouts to `launchpad.net`

The production branch gets rolled out to the web app servers daily, to expose our users to bug fixes not too long after the fix has been verified to work on edge.launchpad.net.

We only do this for the web app servers, since we can update them without any downtime.

XXX: If we're not ready for this, things could get merged into the db branch after being QAed, waiting to get rolled out with the next release.

What about `staging.launchpad.net`

Staging uses the db branch, so that we can do final QA of what gets rolled out to launchpad.net. The stable branch gets automatically merged into the db branch, so that the revisions in the stable branch is testable on staging.