Size: 5252
Comment:
|
← Revision 32 as of 2010-11-08 22:01:23 ⇥
Size: 3902
Comment: rollouts like this are in the future
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
This is a proposal of how a developer branch flows through the different production branches we have, after the developer branch is ready for landing. |
This is a description of how a developer branch flows through the different 'official' branches we have, after the developer branch is ready for landing. |
Line 5: | Line 4: |
= Merge Workflow Draft 2 = | = Merge Workflow = |
Line 7: | Line 6: |
Branches that contain db changes land on the `db` branch, and are QA'd on staging. Bugfixes land on the `stable` branch, and are rolled out daily to a "QA testing" server for QA. Revisions that have been marked QA-OK are copied to a `production` branch, and then rolled out daily to production and edge. |
Branches without db changes (or dependencies on pending db changes) land on the `stable` branch and are rolled out every 30 minutes to a "QA testing" server for QA. Revisions that have been marked QA-OK are [[https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html|eligible]] for deployment. Deployment takes a specific revision of `stable` and deploys that to a named set of machines. The default set is `nodowntime` which encompasses all services that we can deploy to without user visible downtime. |
Line 13: | Line 10: |
{{attachment:merge-workflow-draft-2.jpeg}} | Branches that contain db changes land on the `db-stable` branch, and are QA'd on staging. The `db-stable` branch automatically merges changes from `stable` via a push event in Buildbot. DB changes are rolled out to production and merged back into `stable` once a month. |
Line 15: | Line 15: |
''Image source: [[attachment:merge-workflow-draft-2.jpeg]]'' | {{attachment:mergeworkflow.png}} |
Line 17: | Line 17: |
= Merge Workflow Draft 1 = | ''(Old image: [[attachment:merge-workflow-draft-2.jpeg]])'' |
Line 19: | Line 19: |
To summarize, branches that contain db changes always get landed to the `db` branch. Bug fixes get landed to the `stable` branch, and larger features (consisting of multiple branches) get landed to an integration branch for that feature. For bug fixes, the landed revision stays in the `stable` branch until it has been marked as being QAed. When the revision is marked as good, it automatically gets merged into the `production` branch, which is rolled out to `launchpad.net` daily. For larger features, the integration branch gets combined with stable, and rolled out to `edge.launchpad.net` daily. When the feature is done and QAed, it gets merged into the `stable` branch, and the merged revision finally gets merged into the `production` branch after it has been QAed again on staging. {{attachment:merge-workflow.png}} ''Image source: [[attachment:merge-workflow.dia]]'' |
|
Line 41: | Line 22: |
A revision gets merged from `stable` to `production` only after it has been marked as good in the QA step. This helps ensure that we don't |
A revision gets deployed only after it has been marked as good in the QA step. This helps ensure that we don't |
Line 45: | Line 25: |
In addition to this, a revision only gets merged from `stable` to `production` if all the previous revisions have been marked as good as well. This is because bzr doesn't support cherry picking, and we want to keep track of which revisions have been merged. |
In addition to this, a revision only gets deployed if all the previous revisions have been marked as good as well. This is because cherrypicking can invalidate QA efforts when patches have non-obvious dependencies. |
Line 50: | Line 28: |
XXX: How the QA step works regarding to marking the revision as good/bad is still undefined. |
How the QA step works regarding to marking the revision as good/bad is described on [[QAProcessContinuousRollouts]]. === What if I don't do my QA, or my QA is bad? === If someone does not do their QA, or their branch fails QA, then they will block deployments until that revision is reverted '''and''' the intervening revisions have also been QA'd. Rollouts from `stable` to the QA server will still continue, so other developers can still continue their QA as normal. We assume a two day grace period for QA - if on the third day a branch has not been QA'd, then anyone on the team may revert the offending revision. We must focus on making these reversions as easy and painless as possible. |
Line 54: | Line 42: |
== Feature branches == | === What if I have to do QA that takes a few days? === |
Line 56: | Line 44: |
Features that will involve multiple branches should have an integration branch for that feature. Instead of landing branches related to the feature to `stable`, they land their branches to the integration branch. New revisions in the integration branch will automatically be merged into the `edge` branch. The reason for having an integration branch branch is to keep track of all the revisions that belong to the feature, so that when the feature has been marked as good in the QA process, the whole feature can be rolled out at once. Parts of the feature can still be rolled out by merging specific revisions into `stable`. XXX: How to register new feature branches, and how to merge to them, is still undefined. Maybe a better way would be to use revision properties to tag revisions as being part of a bigger feature, or use bug links? === Resolving conflicts === If there's a conflict when automatically merging the feature branch into the `edge` branch, someone has to manually merge the feature branch into `edge` and resolve the conflicts. |
If a feature requires more than one day of QA, then developers have the option to do the QA on the `staging.launchpad.net` server, to requisition an additional server, (dogfood), or ask for permission to block the daily QA rollouts for more than two or three days. |
Line 79: | Line 50: |
== Bug fixes == | == The Rollout (From december) == |
Line 81: | Line 52: |
Bug fixes for features that are already exposed on `launchpad.net` can go directly to the `stable` branch. After the revision has been QAed, it gets merged into the `production` branch. |
1. Block deployments to all machines 1. lock down devel landings (to release-critical) 1. merge the qa-approved changes from db-stable to devel 1. qastaging will then get the db patches included in it 1. qa up through the db patch merge and nominate the revision to deploy 1. unlock devel landings 1. at the scheduled time do a db deploy of the nominated revision 1. unlock deployments |
Line 85: | Line 61: |
== When to use a feature branch == | == Large Features That Span Branches == |
Line 87: | Line 63: |
So far we've talked about features and bug fixes. However, all features don't need a feature branch. A feature branch is only necessary when you have a larger feature that will require multiple branches to complete. If you have a small feature, or improvement of an existing feature, that you're quite sure will require only one branch, landing it directly on the `stable` branch is fine. |
If you have a feature that will span multiple branches, then it is expected that the developers will create a [[LEP/FeatureFlags|feature switch]] to optionally enable that part of the code, or that early branches that have no user-facing changes will be marked as `qa-untestable`. |
Line 94: | Line 68: |
== Which branch should you branch off from when starting new work? == | == Emergency Fixes/Cherry Picks == |
Line 96: | Line 70: |
When starting new work, you should branch off from the `db` branch if your work will require db changes, and from the `stable` branch otherwise. You shouldn't start of from `edge`, since it contains features that shouldn't be rolled out to production yet, which means that you would have to wait for the existing features to be finished, before your feature can be considered for rollout. |
Either (for confidential non-ZOMG fixes): 1. Block deployments (of the affected servers) 1. push the current deployed revision to prod-devel 1. land on prod-devel 1. deploy from prod stable 1. merge from prod-stable to devel 1. wait for qa blessing for everything outstanding up through the fix rev 1. unblock deployments |
Line 103: | Line 79: |
If possible, the required db changes should go to the `db` branch one cyle in advance, to make it possible to QA the other work on edge. == Continuous rollouts to `launchpad.net` == The `production` branch gets rolled out to the web app servers daily, to expose our users to bug fixes not too long after the fix has been verified to work on `edge.launchpad.net`. We only do this for the web app servers, since we can update them without any downtime. XXX: If we're not ready for this, things could get merged into the `db` branch after being QAed, waiting to get rolled out with the next release. == What about `staging.launchpad.net` == Staging uses the `db` branch, so that we can do final QA of what gets rolled out to `launchpad.net`. The `stable` branch gets automatically merged into the `db` branch, so that the revisions in the `stable` branch is testable on staging. |
Or (for zomg fixes): 1. Block deployments of the affected servers 1. use a cowboy 1. Land on devel 1. wait for qa blessing for everything outstanding up through the fix rev 1. unblock deployments |
This is a description of how a developer branch flows through the different 'official' branches we have, after the developer branch is ready for landing.
Merge Workflow
Branches without db changes (or dependencies on pending db changes) land on the stable branch and are rolled out every 30 minutes to a "QA testing" server for QA. Revisions that have been marked QA-OK are eligible for deployment. Deployment takes a specific revision of stable and deploys that to a named set of machines. The default set is nodowntime which encompasses all services that we can deploy to without user visible downtime.
Branches that contain db changes land on the db-stable branch, and are QA'd on staging. The db-stable branch automatically merges changes from stable via a push event in Buildbot. DB changes are rolled out to production and merged back into stable once a month.
(Old image: merge-workflow-draft-2.jpeg)
The QA step
A revision gets deployed only after it has been marked as good in the QA step. This helps ensure that we don't rollout feature to launchpad.net that haven't been QAed.
In addition to this, a revision only gets deployed if all the previous revisions have been marked as good as well. This is because cherrypicking can invalidate QA efforts when patches have non-obvious dependencies.
How the QA step works regarding to marking the revision as good/bad is described on QAProcessContinuousRollouts.
What if I don't do my QA, or my QA is bad?
If someone does not do their QA, or their branch fails QA, then they will block deployments until that revision is reverted and the intervening revisions have also been QA'd. Rollouts from stable to the QA server will still continue, so other developers can still continue their QA as normal.
We assume a two day grace period for QA - if on the third day a branch has not been QA'd, then anyone on the team may revert the offending revision.
- We must focus on making these reversions as easy and painless as possible.
What if I have to do QA that takes a few days?
If a feature requires more than one day of QA, then developers have the option to do the QA on the staging.launchpad.net server, to requisition an additional server, (dogfood), or ask for permission to block the daily QA rollouts for more than two or three days.
The Rollout (From december)
- Block deployments to all machines
- lock down devel landings (to release-critical)
- merge the qa-approved changes from db-stable to devel
- qastaging will then get the db patches included in it
- qa up through the db patch merge and nominate the revision to deploy
- unlock devel landings
- at the scheduled time do a db deploy of the nominated revision
- unlock deployments
Large Features That Span Branches
If you have a feature that will span multiple branches, then it is expected that the developers will create a feature switch to optionally enable that part of the code, or that early branches that have no user-facing changes will be marked as qa-untestable.
Emergency Fixes/Cherry Picks
Either (for confidential non-ZOMG fixes):
- Block deployments (of the affected servers)
- push the current deployed revision to prod-devel
- land on prod-devel
- deploy from prod stable
- merge from prod-stable to devel
- wait for qa blessing for everything outstanding up through the fix rev
- unblock deployments
Or (for zomg fixes):
- Block deployments of the affected servers
- use a cowboy
- Land on devel
- wait for qa blessing for everything outstanding up through the fix rev
- unblock deployments