launchpad development
A user's guide to QA for Continuous Rollouts
This page is a quick guide to landing your code. You need to follow this process to keep our system of continuous rollouts flowing.
The Really Important Part
Deployable must be kept up to date with the QA status of each branch that has landed and passed buildbot.
As a developer you have the following responsibilities:
- It is your responsibility to set the QA status of your branches properly so that we do not deploy untested code.
- It is your responsibility to QA your code in a timely manner so that your QA does not block the continuous rollout of everyone's code.
*It is really important that you set the QA status correctly because your code will soon be in production!*
Changes to 'db-stable' use a separate deployable project, though are typically rolled out less frequently as DB rollouts require (brief) downtime.
Landing your branch
If the branch fixes bugs, make sure to link them to the merge proposal. This helps people work out which bugs to close when deploying changes to production.
Top-approve the merge proposal to land it (i.e. change the "Status" field to "Approved"). If the merge proposal has appropriate votes, and if it has no conflicts, a Jenkins job will merge it shortly afterwards.
Wait for the commit to pass buildbot, fixing or retrying as needed. Once it passes buildbot, it will be merged onto the stable branch, and deployable will update shortly afterwards.
If your branch does not need QA (for example, simple documentation fixes, test fixes, or code refactorings usually do not need QA), then you can immediately mark it as "Yes" on deployable.
If your branch needs QA (as is the case for most feature work), it is now your responsibility to check whether it passes. This does not necessarily mean that it fixes the problem you were trying to fix; the standard for QA is that the change must not introduce regressions, since that governs whether it can safely be deployed to production. You should normally focus on things that may not be as well-exercised by the test suite, such as integration between multiple services.
Find a non-production instance of Launchpad that has the necessary facilities for you to test your change: webservice API and UI changes can normally be tested on qastaging, while other changes may require using staging or dogfood. (Consult your colleagues if unsure.) qastaging updates automatically shortly after changes land on stable; staging updates automatically shortly after changes land on db-stable; dogfood is updated manually by developers. Do whatever tests you need to be confident in your change, and mark the commit as deployable or not based on your testing.
Backing out a commit
What you do:
Back out the code by reversing a patch or using git revert; or, if appropriate, fix the code in place. (Be careful of the latter approach; we want to avoid ending up in a situation where production deployments are blocked on repeated attempts to fix broken code.)
- Commit your change, linking to each bug the original branch was linked.
- Create a merge proposal and get it reviewed, or review it yourself.
- Top-approve the merge proposal to land the change, and wait for it to be merged.
If your previous change failed to pass buildbot, you will now need to force a build to get buildbot to try again.
- Wait for your new change to pass buildbot and appear on deployable.
- Mark your old change as "No" and "Rolled back by" your new change on deployable.
Deployable will keep track of the longest prefix of the deployment queue containing only commits that are themselves deployable or that have been rolled back by other commits which are deployable.
