• Name: Pre Release QA Process (Using Launchpad)

• Owner: Launchpad Team

• Effective: Under experiment as of 2009-10-01.

• Experiment: Will replace Pre Release QA Process

• Review: 2009-12-01

Using Launchpad as main QA tool

Why are we doing this?

• Improve how we do QA today, by:
  • Being able to search items dynamically using Launchpad search and Launchpad API
  • Not forcing developers to edit the wiki pages to do their QA
• To reduce/eliminate the number of orphaned commits per cycle.
• To make easier and more reliable to gather stats of the cycle

What are we doing?

• Eating our own dogfood. The experiment consists in using Launchpad to know which bugs are yet to be tested, and to test them using the information we already have in the Merge Proposals (now shown in the bug page). It's easier to use all that organized and aggregated information than use some parallel that contains duplicated data. Also it's much easier to change a status of a bug from untested to tested than editing the Test Plan wiki page.

What are the risks?

• Leave untested items behind;
• By ignoring orphaned commits, assuming they don't have the need to be tested, it's possible to miss items to test because people forget to add bug numbers to their commit messages.
  • The script tries to workaround this by trying to identify the branch that originated that revision, finding on Launchpad its linked bugs. We need to guarantee that people will link bugs to branches for this to be successful.
• Face limitations of Launchpad search that creates barriers instead of removing them;
• Testing not properly, considering we're using the instructions in the MP and this could be incomplete.
  • We can workaround this by assuring people will detail their tests well in the MP, and if not, do that in the bug itself.

How will we know if we were successful?

• The intention is to make QA easier by using only Launchpad and removing the need of editing wiki pages, so we'll know we're successful when we have few (or none) items pending to test during the cycle.

Process under experiment

• Running an automated script that:
  • Watches the main development branches analysing the new commits one by one, as the current test-plan tool does;
  • For each revision:
    • Tries to find out the bugs related to that commit;
    • If found it, in the commit message or as linked bugs in the branch in Launchpad, does the following:
      • Marks the mentioned bug(s) as Fix Committed;
      • Adds a comment to it documenting which commit was responsible for that status change;
      • Adds the qa-needstesting tag, if it has no qa tags; If another qa tag has been found, removes it and adds the qa-needstesting one.
      • Assign the bug to the committer, if the bug has no assignees.
      • Assign the bug to the current milestone, to guarantee the burndown script will find only bugs related to the present cycle.
    • If the commit is a testfix or mentions no bugs, we consider it an orphan commit and move on.
• The search for items to QA to a given person can be done through an advanced bug search in Launchpad:
  • with Fix Committed or Fix Released status;
  • with qa-needstesting tag;
  • that are assigned to the person;
  • Don't hide duplicates (some bugs fixed may be dupes of others and not show as pending).

It's possible to search a person qa-needstesting bugs through a Launchpad search like this: https://bugs.edge.launchpad.net/launchpad-project/+bugs?field.status%3Alist=FIXCOMMITTED&field.status%3Alist=FIXRELEASED&assignee_option=choose&field.omit_dupes.used=&field.tag=qa-needstesting&field.assignee=ursinha .

Comments

• Launchpad doesn't have in its UI a way to do a search so we'd get the same results we have in the testplan page, that are all items of all members of that team. We only can search items per team, per tag or per assignee (one at a time). So if someone commits a fix for a bug of other team it will only show up in the search for his own bugs, and not the team's (project). I've filed a bug for that: https://bugs.edge.launchpad.net/malone/+bug/520569.

• Testfixes need to be separated commits. They're ignored, so should be really only testfixes. Please, do not commit bug fixes along with testfixes.
• Attention when writing your commit message! If you use ec2test script, it should automatically add the bug number to the commit message. If you don't, and you forget to add it manually, the script will try to find out if the bug number is mentioned in the merged branch nick or if the merged branch has linked bugs. If everything fails, bzzt, your commit goes to /dev/null. So, if your branch needs QA, be sure to add the bug numbers, all of them, to the commit message, and link them to your branch in Launchpad. Otherwise it will fall into oblivion. The script isn't creating bugs to orphaned commits because we QA bugs, not branches, and by creating bugs we add the need of triaging useless bugs, when (we hope) most orphaned commits in fact don't need any QA.

One alternative to that is always adding the tag bug to the commit message, but having it with value None (as we do with ui) when no QA is needed.