launchpad development|
Size: 3049
Comment: Fix link.
|
Size: 4022
Comment: Rewrite based on feedback from many.
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| = Quality Assurance: Verifying Changes to Launchpad = | = Quality Assurance for Launchpad Changes = |
| Line 3: | Line 3: |
| ||<tablestyle="width: 100%;" colspan=3 style="background: #2a2929; font-weight: bold; color: #f6bc05;">This page describes how to get changes to Launchpad verified ("QA'd") before they are released. || | ||<tablestyle="width: 100%;" colspan=3 style="background: #2a2929; font-weight: bold; color: #f6bc05;">This page describes how to get changes to Launchpad QA'd before they are released. || |
| Line 7: | Line 7: |
| First, the author of the change should have already described the QA procedure in the [[CoverLetters| merge proposal cover letter]]. The ideal QA procedure is runnable by any Launchpad user, not just developers. It's best if a change can be QA'd using [[https://edge.launchpad.net/|edge]] or [[https://staging.launchpad.net/|staging]], not a local `launchpad.dev` development instance (see Barry Warsaw's [[https://lists.launchpad.net/launchpad-dev/msg00292.html|mail about this]]). This should be the case for most changes, though there are occasional exceptions. Next, the author needs to find someone to actually do that QA... |
'''''QA''''' is slightly different from '''''verification'''''. Verification is about making sure the change does what it claims to do: if it fixes a bug, then the bug should no longer be reproducible. The author of a change should do verification before and after the change lands. But QA goes beyond that: QA is about actively trying to break things. For this reason, the ideal QA tester is someone other than the change's author, though it's okay for the author to do QA if no one else is available right away. |
| Line 13: | Line 11: |
| People are willing to do QA, but generally need to be asked to do it for a specific change -- consider that request to be part of finishing the change. Volunteers rarely show up on their own initiative looking for random changes to QA; instead, the author of a change needs to find people and explicitly request them to QA it. | Hop into IRC ([[irc://irc.freenode.net/launchpad-dev|#launchpad-dev]] and [[irc://irc.freenode.net/launchpad|#launchpad]] at irc.freenode.net) and see if anyone is available right now to do QA on your change. If you know people there, ping them by name. |
| Line 15: | Line 13: |
| The best way to find them is to look at the discussion around the fix, especially in the bug or mailing list thread that preceded the fix. Are there people who made comments, or who said that they were looking forward to the improvement in question? If so, they could be asked to QA it, and to send their feedback in the same forum (and if they can update the [[#pending-qa pending QA wiki page]], that's even better). | If you can't find anyone in real time, then QA your change yourself (see below for how), and ''then'' try to get someone else for a second sign-off. It's important not to let QA become a bottleneck; it needs to get done so the change can be deployed. |
| Line 17: | Line 15: |
| If there are no obvious candidates to QA a change, the author should go into IRC ([[irc://irc.freenode.net/launchpad-dev|#launchpad-dev IRC channel]] at irc.freenode.net) and try to find someone there. If no takers, it's fine to try the user forum ([[irc://irc.freenode.net/launchpad-dev|#launchpad]]) too, and then try posting on [[https://launchpad.net/~launchpad-dev|launchpad-dev@]] or [[https://launchpad.net/~launchpad-users|launchpad-users@]]. The subject line of the mail should specify the nature of the change being QA'd, so people interested in that change are likely to see it (and so everyone else can skip it). | But it's still good to have a second sign-off, and you can often find someone to do it by looking at the discussion around the change, especially in the relevant bug or mailing list thread. Are there people who made comments, or who said that they were looking forward to the change? If so, see if they're willing to do QA, and to post their feedback in the same forum (or if they can update the [[#pending-qa pending QA wiki page]], that's even better). == Don't Give Too Much Instruction == In general, the author shouldn't give detailed instructions about the change. Instead, just lead the QA person to where the change is and see if they can figure out how the new interface (if any) works, and if they can break it. An exception is when the change needs detailed setup to be tested at all -- in that case, describe the procedures in the [[CoverLetters|merge proposal cover letter]] (for example, see the "Q/A" section of [[https://code.edge.launchpad.net/~kfogel/launchpad/old-1.6-ml-archiver-ui/+merge/6001|this merge proposal]]). It's best if a change can be QA'd using [[https://edge.launchpad.net/|edge]] or [[https://staging.launchpad.net/|staging]], not a local `launchpad.dev` development instance (see Barry Warsaw's [[https://lists.launchpad.net/launchpad-dev/msg00292.html|mail about this]]). The ideal QA procedure is runnable by any Launchpad user, not just by developers. <<Anchor(recording)>> == Recording QA Procedures == When doing QA, please record what the various ways you tried to break things. It doesn't have to be at a mouse-click-by-mouse-click level of detail; just give enough description so another person can tell if any important areas have been left uncovered. (This is especially important when the author of the change is doing QA, because it's easy to let unconscious knowledge of the intent behind the change influence how one tests the change -- describing your QA procedure is a good way of keeping yourself honest.) |
| Line 20: | Line 29: |
| == Listing Changes That Still Need QA == | == List of Changes That Still Need QA == |
Quality Assurance for Launchpad Changes
This page describes how to get changes to Launchpad QA'd before they are released. |
When a new change lands Launchpad, the change isn't finished yet -- it still needs to be QA'd.
QA is slightly different from verification. Verification is about making sure the change does what it claims to do: if it fixes a bug, then the bug should no longer be reproducible. The author of a change should do verification before and after the change lands. But QA goes beyond that: QA is about actively trying to break things. For this reason, the ideal QA tester is someone other than the change's author, though it's okay for the author to do QA if no one else is available right away.
How to Get QA for a Change
Hop into IRC (#launchpad-dev and #launchpad at irc.freenode.net) and see if anyone is available right now to do QA on your change. If you know people there, ping them by name.
If you can't find anyone in real time, then QA your change yourself (see below for how), and then try to get someone else for a second sign-off. It's important not to let QA become a bottleneck; it needs to get done so the change can be deployed.
But it's still good to have a second sign-off, and you can often find someone to do it by looking at the discussion around the change, especially in the relevant bug or mailing list thread. Are there people who made comments, or who said that they were looking forward to the change? If so, see if they're willing to do QA, and to post their feedback in the same forum (or if they can update the #pending-qa pending QA wiki page, that's even better).
Don't Give Too Much Instruction
In general, the author shouldn't give detailed instructions about the change. Instead, just lead the QA person to where the change is and see if they can figure out how the new interface (if any) works, and if they can break it. An exception is when the change needs detailed setup to be tested at all -- in that case, describe the procedures in the merge proposal cover letter (for example, see the "Q/A" section of this merge proposal).
It's best if a change can be QA'd using edge or staging, not a local launchpad.dev development instance (see Barry Warsaw's mail about this). The ideal QA procedure is runnable by any Launchpad user, not just by developers.
Recording QA Procedures
When doing QA, please record what the various ways you tried to break things. It doesn't have to be at a mouse-click-by-mouse-click level of detail; just give enough description so another person can tell if any important areas have been left uncovered. (This is especially important when the author of the change is doing QA, because it's easy to let unconscious knowledge of the intent behind the change influence how one tests the change -- describing your QA procedure is a good way of keeping yourself honest.)
List of Changes That Still Need QA
Prior to any release or milestone, there is always a list of changes needing QA. For example, here is the list of changes needing QA leading up to the 2.2.8 milestone.
Once someone has tested a change, they should move it to the "OK" or "BAD" section on that page as appropriate, and follow up with the original author and/or in a bug or mailing list thread as appropriate. If you know a change has been tested (because you saw discussion elsewhere about it), please just update the QA page as needed, based on that discussion.
