Quality Assurance for Launchpad Changes

When a new change lands Launchpad, the change isn't finished yet -- it still needs to be QA'd.

QA is slightly different from verification. Verification is about making sure the change does what it claims to do: if it fixes a bug, then the bug should no longer be reproducible. The author of a change should do verification before and after the change lands. But QA goes beyond that: QA is about actively trying to break things. For this reason, the ideal QA tester is someone other than the change's author, though it's okay for the author to do QA if no one else is available right away.

How to Get QA for a Change

Hop into IRC (#launchpad-dev and #launchpad at irc.libera.chat) and see if anyone is available right now to do QA on your change. If you know people there, ping them by name.

If you can't find anyone in real time, then QA your change yourself (see below for how), and then try to get someone else for a second sign-off. It's important not to let QA become a bottleneck; it needs to get done so the change can be deployed.

But it's still good to have a second sign-off, and you can often find someone to do it by looking at the discussion around the change, especially in the relevant bug or mailing list thread. Are there people who made comments, or who said that they were looking forward to the change? If so, see if they're willing to do QA, and to post their feedback in the same forum (or if they can update the #pending-qa pending QA wiki page, that's even better).

Don't Give Too Much Instruction

In general, the author shouldn't give detailed instructions about the change. Instead, just lead the QA person to where the change is and see if they can figure out how the new interface (if any) works, and if they can break it. An exception is when the change needs detailed setup to be tested at all -- in that case, describe the procedures in the merge proposal cover letter (for example, see the "Q/A" section of this merge proposal).

It's best if a change can be QA'd using edge or staging, not a local launchpad.dev development instance (see Barry Warsaw's mail about this). The ideal QA procedure is runnable by any Launchpad user, not just by developers.

Recording QA Procedures

When doing QA, please record what the various ways you tried to break things. It doesn't have to be at a mouse-click-by-mouse-click level of detail; just give enough description so another person can tell if any important areas have been left uncovered. (This is especially important when the author of the change is doing QA, because it's easy to let unconscious knowledge of the intent behind the change influence how one tests the change -- describing your QA procedure is a good way of keeping yourself honest.)

List of Changes That Still Need QA

Prior to any release or milestone, there is always a list of changes needing QA. For example, here is the list of changes needing QA leading up to the 2.2.8 milestone.

Once someone has tested a change, they should move it to the "OK" or "BAD" section on that page as appropriate, and follow up with the original author and/or in a bug or mailing list thread as appropriate. If you know a change has been tested (because you saw discussion elsewhere about it), please just update the QA page as needed, based on that discussion.