launchpad development|
Size: 5247
Comment: Oh dear, how long have those been wrong?
|
← Revision 13 as of 2021-11-03 14:33:32 ⇥
Size: 4256
Comment: Rework to describe modern git-based process
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 4: | Line 4: |
| This page is a quick guide to landing your code using the new QA flags and process. You need to follow this process to keep our new system of continuous rollouts flowing. |
This page is a quick guide to landing your code. You need to follow this process to keep our system of continuous rollouts flowing. |
| Line 10: | Line 8: |
| Every branch that lands in our merge workflow has to have some sort of QA status associated with it, like 'needs testing' or 'does not require QA'. When you land a branch, you need to pass some switches into the 'bzr lp-land' and 'ec2 land' commands to tell them what the QA for your branch should look like. The land commands will set the commit message for your branch so that our automated QA tools know what to do next. |
[[https://deployable.ols.canonical.com/project/launchpad|Deployable]] must be kept up to date with the QA status of each branch that has landed and passed buildbot. |
| Line 27: | Line 20: |
| Robert is regularly using cherry picks to roll out every revision of 'stable' that has passed QA. *It is really important that you set the QA status |
*It is really important that you set the QA status |
| Line 31: | Line 23: |
| Changes to 'db-stable' are not part of this new process. Rollout of the db-stable branch will take place as normal at the end of the cycle. |
Changes to 'db-stable' use a [[https://deployable.ols.canonical.com/project/launchpad-db|separate deployable project]], though are typically rolled out less frequently as DB rollouts require (brief) downtime. |
| Line 37: | Line 28: |
| There are a few different situations you will meet in your day-to-day development cycle and we have a set of QA tags that are appropriate to each. Below is a list of the most common situations and how you set the QA status appropriately. |
If the branch fixes bugs, make sure to [[https://help.launchpad.net/Code/Git#Linking_to_bugs|link them to the merge proposal]]. This helps people work out which bugs to close when deploying changes to production. Top-approve the merge proposal to land it (i.e. change the "Status" field to "Approved"). If the merge proposal has appropriate votes, and if it has no conflicts, a [[https://jenkins.ols.canonical.com/online-services/job/launchpad/|Jenkins job]] will merge it shortly afterwards. Wait for the commit to pass [[http://lpbuildbot.canonical.com/waterfall|buildbot]], fixing or retrying as needed. Once it passes buildbot, it will be merged onto the stable branch, and deployable will update shortly afterwards. If your branch does not need QA (for example, simple documentation fixes, test fixes, or code refactorings usually do not need QA), then you can immediately mark it as "Yes" on deployable. If your branch needs QA (as is the case for most feature work), it is now your responsibility to check whether it passes. This does not necessarily mean that it fixes the problem you were trying to fix; the standard for QA is that the change must not introduce regressions, since that governs whether it can safely be deployed to production. You should normally focus on things that may not be as well-exercised by the test suite, such as integration between multiple services. Find a [[https://wiki.canonical.com/Launchpad/Instances|non-production instance of Launchpad]] that has the necessary facilities for you to test your change: webservice API and UI changes can normally be tested on qastaging, while other changes may require using staging or dogfood. (Consult your colleagues if unsure.) qastaging updates automatically shortly after changes land on stable; staging updates automatically shortly after changes land on db-stable; dogfood is updated manually by developers. Do whatever tests you need to be confident in your change, and mark the commit as deployable or not based on your testing. |
| Line 43: | Line 41: |
| == The branch fixes one or more bugs, and it needs QA == | == Backing out a commit == |
| Line 47: | Line 45: |
| 1. Link the bug(s) to the branch with --fixes or in the web UI. 2. Use 'ec2 land' or 'bzr lp-land' as normal. 3. QA as normal (tag the bug as qa-ok, qa-bad, or qa-untestable). |
1. Back out the code by reversing a patch or using `git revert`; or, if appropriate, fix the code in place. (Be careful of the latter approach; we want to avoid ending up in a situation where production deployments are blocked on repeated attempts to fix broken code.) 1. Commit your change, linking to each bug the original branch was linked. 1. Create a merge proposal and get it reviewed, or review it yourself. 1. Top-approve the merge proposal to land the change, and wait for it to be merged. 1. If your previous change failed to pass buildbot, you will now need to [[http://lpbuildbot.canonical.com/force|force a build]] to get buildbot to try again. 1. Wait for your new change to pass buildbot and appear on deployable. 1. Mark your old change as "No" and "Rolled back by" your new change on deployable. |
| Line 51: | Line 53: |
| The bug(s) will be tagged qa-needstesting. After doing your QA you must mark the bug(s) as qa-ok, qa-bad, or qa-untestable. More below about what each QA tag does. == The branch does NOT need QA == What you do: 1. If there is a bug for this branch, link it; if not, that is fine too. 2. Use 'ec2 land --no-qa' or 'bzr lp-land --no-qa'. The bug will be tagged as 'qa-untestable', and the branch will be deployed. The bug will be closed. == The branch is one in a series, and it needs QA == What you do: 1. Link the bug. 2. Use 'ec2 land --incremental' or 'bzr lp-land --incremental' to land the branch. 3. QA as normal. The bug will remain open. == The branch is one in a series, but does NOT need QA == What you do: 1. Link the bug. 2. Use 'ec2 land --no-qa --incremental' or 'bzr lp-land --no-qa --incremental'. The branch will be deployed. The bug will remain open. = The qa-ok, qa-untestable, and qa-bad tags = To finish normal QA for a branch you must choose one of three tags: qa-ok, qa-untestable, or qa-bad. If you use qa-ok or qa-untestable, then your branch will be deployed. If you use the qa-bad tag, then you have to back your code out of the mainline. '''Bugs marked as qa-bad or qa-needstesting block the rollout of every revision that comes after them!''' == Backing out a revision == What you do: 1. Back out the code by reversing a patch or 'bzr merge -r Y..X'. For instance if rev 101 needs to be reverted you would reverse it with 'bzr merge -r 101..100'. 1. Commit your change, linking to each bug the original branch was linked. (NOTE: using multiple --fixes=lp:bug_number on the commit does not seem to work.) 1. Create a merge proposal and get it reviewed, or review it yourself. 1. Land the code using 'ec2 land --rollback=BADREVNO --testfix' or 'bzr lp-land --rollback=BADREVNO --testfix'. In our example, 'ec2 land --rollback=101 --testfix' or 'bzr lp-land --rollback=101 --testfix'. --testfix is not required for the backout but you're probably doing it because buildbot is in testfix mode. The bug will be tagged as bad-commit-*, all the qa-* tags will be removed, and all revisions up to and including the rollback revision will be deployed. You may also want to use this procedure to back out untested code that is blocking too many other user's deployments. This will become more of an issue when we start doing daily rollouts. = Has my branch been deployed yet? = We have created HTML reports that shows the QA status of all of the revisions on stable and db-stable: * http://lpqateam.canonical.com/qa-reports/deployment-stable.html * http://lpqateam.canonical.com/qa-reports/deployment-db-stable.html The reports show whose change is blocking rollouts, links to the blocking Bug, and a list of all undeployed revisions that have passed QA. The reports update every 20 minutes. If you do not see your revision on the list, then it it is either blocked by someone else, or it has already been deployed. = Who to contact if you have questions = If you have any questions about the new process please feel free to contact anyone on the Launchpad Foundations team (Ursinha, matsubara, or mars). |
Deployable will keep track of the longest prefix of the deployment queue containing only commits that are themselves deployable or that have been rolled back by other commits which are deployable. |
A user's guide to QA for Continuous Rollouts
This page is a quick guide to landing your code. You need to follow this process to keep our system of continuous rollouts flowing.
The Really Important Part
Deployable must be kept up to date with the QA status of each branch that has landed and passed buildbot.
As a developer you have the following responsibilities:
- It is your responsibility to set the QA status of your branches properly so that we do not deploy untested code.
- It is your responsibility to QA your code in a timely manner so that your QA does not block the continuous rollout of everyone's code.
*It is really important that you set the QA status correctly because your code will soon be in production!*
Changes to 'db-stable' use a separate deployable project, though are typically rolled out less frequently as DB rollouts require (brief) downtime.
Landing your branch
If the branch fixes bugs, make sure to link them to the merge proposal. This helps people work out which bugs to close when deploying changes to production.
Top-approve the merge proposal to land it (i.e. change the "Status" field to "Approved"). If the merge proposal has appropriate votes, and if it has no conflicts, a Jenkins job will merge it shortly afterwards.
Wait for the commit to pass buildbot, fixing or retrying as needed. Once it passes buildbot, it will be merged onto the stable branch, and deployable will update shortly afterwards.
If your branch does not need QA (for example, simple documentation fixes, test fixes, or code refactorings usually do not need QA), then you can immediately mark it as "Yes" on deployable.
If your branch needs QA (as is the case for most feature work), it is now your responsibility to check whether it passes. This does not necessarily mean that it fixes the problem you were trying to fix; the standard for QA is that the change must not introduce regressions, since that governs whether it can safely be deployed to production. You should normally focus on things that may not be as well-exercised by the test suite, such as integration between multiple services.
Find a non-production instance of Launchpad that has the necessary facilities for you to test your change: webservice API and UI changes can normally be tested on qastaging, while other changes may require using staging or dogfood. (Consult your colleagues if unsure.) qastaging updates automatically shortly after changes land on stable; staging updates automatically shortly after changes land on db-stable; dogfood is updated manually by developers. Do whatever tests you need to be confident in your change, and mark the commit as deployable or not based on your testing.
Backing out a commit
What you do:
Back out the code by reversing a patch or using git revert; or, if appropriate, fix the code in place. (Be careful of the latter approach; we want to avoid ending up in a situation where production deployments are blocked on repeated attempts to fix broken code.)
- Commit your change, linking to each bug the original branch was linked.
- Create a merge proposal and get it reviewed, or review it yourself.
- Top-approve the merge proposal to land the change, and wait for it to be merged.
If your previous change failed to pass buildbot, you will now need to force a build to get buildbot to try again.
- Wait for your new change to pass buildbot and appear on deployable.
- Mark your old change as "No" and "Rolled back by" your new change on deployable.
Deployable will keep track of the longest prefix of the deployment queue containing only commits that are themselves deployable or that have been rolled back by other commits which are deployable.
