launchpad development|
Size: 1943
Comment:
|
← Revision 17 as of 2017-03-30 14:19:02 ⇥
Size: 5133
Comment: update qa-reports links
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ## page was renamed from PolicyAndProcess/PreReleaseQAProcess ## page was renamed from PolicyAndProcess/PreReleaseQAProcess/Experiment |
|
| Line 3: | Line 5: |
| * '''Name:''' Pre Release QA Process (Using Launchpad) | * '''Name:''' QA Process |
| Line 5: | Line 7: |
| * '''Effective:''' Under experiment as of 2009-10-01. * '''Experiment:''' Will replace [[PolicyAndProcess/PreReleaseQAProcess|Pre Release QA Process]] * '''Review:''' 2009-12-01 |
* '''Effective:''' March 2010 |
| Line 10: | Line 10: |
| = Using Launchpad as main QA tool = | = Launchpad is the main QA Tool for Launchpad = |
| Line 14: | Line 14: |
| * Improve how we do QA today, by: * Being able to search items dynamically using Launchpad search and Launchpad API * Not forcing developers to edit the wiki pages to do their QA * To reduce/eliminate the number of orphaned commits per cycle. * To make easier and more reliable to gather stats of the cycle |
* Launchpad provides a solid base for managing QA using bug tags. * Launchpad also lets us gather statistics on velocity and QA status pretty easily. |
| Line 22: | Line 19: |
| * Running an automated script that watches the main development branches analysing the commits one by one, as the current test-plan tool does, but instead of adding all items to the wiki test plans, it marks the mentioned bug as Fix Committed in Launchpad, adds a comment to it documenting which commit was responsible for that status change, and adds the qa-needstesting tag. If the commit is an orphaned one, it adds to the team wiki page as a fallback. * All the orphaned commits need to be handled by creating one bug for it. The goal is to eliminate orphaned commits per cycle. * The search for items to QA to a given person can be done through an advanced bug search in Launchpad: * with Fix Committed or Fix Released status * with qa-needstesting tag * that are assigned to you * Don't hide duplicates (some bugs fixed may be dupes of others and not show as pending) |
* We record qa status for individual commits as tags against bugs in Launchpad. * Special instructions for QA are put into the merge proposal when landing a branch. |
| Line 32: | Line 24: |
| * | * Not all commits have bugs, and some bugs require multiple commits. This is a bit awkward and developers are particularly sensitive to such friction. We tolerate it for now. * Orphaned commits - merges from stable to db-devel and from db-stable to devel - may introduce regressions but are not QA'd at present. * Developers may mis-assess the risk of a commit and mark it as not needing QA when it does actually need it. |
| Line 34: | Line 28: |
| == How will we know if we were successful? == | == How successful are we? == |
| Line 36: | Line 30: |
| * | * We regularly catch issues (1/week-or-two) on qastaging and staging. * Infrequently (1/month) we deploy broken code and have to revert. (reverting is cheap except for db deploys). * Recording the result of QA is easy, but some corner cases regularly confuse developers leading. |
| Line 40: | Line 36: |
| * | N/A - no current experiment. === The tags === Bugs can have one of the five qa tags: * qa-needstesting: the script should add this tag when the bug is ready to be QAed. Don't add it manually! It's the only tag you shouldn't care about setting. From the moment the bug has this tag on, you're allowed to change for one of the other tags. * qa-ok: when your bug is QAed and OK, just remove the qa-needstesting and add a qa-ok. The same as moving your old test item from the NEEDSTESTING section to the OK one. * qa-bad: similarly to qa-ok, but when your bug isn't working as expected, of course. :) * qa-untestable: to those items that can't be QAed for some reason. I still don't know how useful this tag is, given all items we cannot test shouldn't have bugs. * qa-rcfixed: when a bug is fixed in RC mode. The bugs for bad commits also have * bad-commit-NNNNN to record the bad commit, and a matching landing with [rollback=NNNN] to record the revision that fixes (either by rolling back, or by correcting) the bad commit. === qa-tagger === qa-tagger is a script running from ~lpqateam on devpad which monitors the main Launchpad branches, and handles the new revisions as explained below: * For each new merged revision: * Checks if the commit is a testfix or a buildbot commit. Those are automatically ignored. [1] * If not, tries to find out the bugs related to that commit: * By looking in the commit message; * By looking at the branch nick; * By fetching the original merged branch in Launchpad, examining if there are linked bugs. * If bugs were found: * Marks them as Fix Committed; * Adds a comment in the bugs, documenting which commit was responsible for that status change; * Adds the "qa-needstesting" tag. If another qa tag has been found, removes it and adds the qa-needstesting one. It assumes the commit is trying to fix the bug again, so should be re-tested. * Assigns the bug to the committer, if the bug has no assignees. * Assigns the bug to the current milestone, to guarantee the burndown script will find only bugs related to the present cycle. * If no bugs are found, the commit is considered orphaned, and ignored.[2] * To see what QA is needed, visit * stable: [http://lpqateam.canonical.com/qa-reports/deployment-stable.html] * db-stable: [http://lpqateam.canonical.com/qa-reports/deployment-db-stable.html] [1] Testfixes need to be separated commits. They're ignored, so should be really only testfixes. Do not commit bug fixes along with testfixes. [2] Be careful when writing your commit message. If you use ec2test script, it should automatically add the bug number to the commit message. If you don't, and you forget to add it manually, the script will try to find out if the bug number is mentioned in the merged branch nick or if the merged branch has linked bugs. If everything fails, bzzt, your commit goes to /dev/null. So, if your branch needs QA, be sure to add the bug numbers, all of them, to the commit message, and link them to your branch in Launchpad, otherwise it will fall into oblivion. The script isn't creating bugs to orphaned commits because we QA bugs, not branches, and by creating bugs for each branch we're possibly filing a bunch of invalid bugs, that will take some time to be triaged. (Hopefully) most orphaned commits in fact don't need any QA. |
Name: QA Process
Owner: Launchpad Team
Effective: March 2010
Launchpad is the main QA Tool for Launchpad
Why are we doing this?
- Launchpad provides a solid base for managing QA using bug tags.
- Launchpad also lets us gather statistics on velocity and QA status pretty easily.
What are we doing?
- We record qa status for individual commits as tags against bugs in Launchpad.
- Special instructions for QA are put into the merge proposal when landing a branch.
What are the risks?
- Not all commits have bugs, and some bugs require multiple commits. This is a bit awkward and developers are particularly sensitive to such friction. We tolerate it for now.
- Orphaned commits - merges from stable to db-devel and from db-stable to devel - may introduce regressions but are not QA'd at present.
- Developers may mis-assess the risk of a commit and mark it as not needing QA when it does actually need it.
How successful are we?
- We regularly catch issues (1/week-or-two) on qastaging and staging.
- Infrequently (1/month) we deploy broken code and have to revert. (reverting is cheap except for db deploys).
- Recording the result of QA is easy, but some corner cases regularly confuse developers leading.
Process under experiment
N/A - no current experiment.
The tags
Bugs can have one of the five qa tags:
- qa-needstesting: the script should add this tag when the bug is ready to be QAed. Don't add it manually! It's the only tag you shouldn't care about setting. From the moment the bug has this tag on, you're allowed to change for one of the other tags.
- qa-ok: when your bug is QAed and OK, just remove the qa-needstesting and add a qa-ok. The same as moving your old test item from the NEEDSTESTING section to the OK one.
qa-bad: similarly to qa-ok, but when your bug isn't working as expected, of course.
- qa-untestable: to those items that can't be QAed for some reason. I still don't know how useful this tag is, given all items we cannot test shouldn't have bugs.
- qa-rcfixed: when a bug is fixed in RC mode.
The bugs for bad commits also have
- bad-commit-NNNNN to record the bad commit, and a matching landing with [rollback=NNNN] to record the revision that fixes (either by rolling back, or by correcting) the bad commit.
qa-tagger
qa-tagger is a script running from ~lpqateam on devpad which monitors the main Launchpad branches, and handles the new revisions as explained below:
- For each new merged revision:
- Checks if the commit is a testfix or a buildbot commit. Those are automatically ignored. [1]
- If not, tries to find out the bugs related to that commit:
- By looking in the commit message;
- By looking at the branch nick;
- By fetching the original merged branch in Launchpad, examining if there are linked bugs.
- If bugs were found:
- Marks them as Fix Committed;
- Adds a comment in the bugs, documenting which commit was responsible for that status change;
- Adds the "qa-needstesting" tag. If another qa tag has been found, removes it and adds the qa-needstesting one. It assumes the commit is trying to fix the bug again, so should be re-tested.
- Assigns the bug to the committer, if the bug has no assignees.
- Assigns the bug to the current milestone, to guarantee the burndown script will find only bugs related to the present cycle.
- If no bugs are found, the commit is considered orphaned, and ignored.[2]
- To see what QA is needed, visit
[1] Testfixes need to be separated commits. They're ignored, so should be really only testfixes. Do not commit bug fixes along with testfixes.
[2] Be careful when writing your commit message. If you use ec2test script, it should automatically add the bug number to the commit message. If you don't, and you forget to add it manually, the script will try to find out if the bug number is mentioned in the merged branch nick or if the merged branch has linked bugs. If everything fails, bzzt, your commit goes to /dev/null. So, if your branch needs QA, be sure to add the bug numbers, all of them, to the commit message, and link them to your branch in Launchpad, otherwise it will fall into oblivion. The script isn't creating bugs to orphaned commits because we QA bugs, not branches, and by creating bugs for each branch we're possibly filing a bunch of invalid bugs, that will take some time to be triaged. (Hopefully) most orphaned commits in fact don't need any QA.
