Diff for "QAProcessContinuousRollouts"

Not logged in - Log In / Register

Differences between revisions 1 and 2
Revision 1 as of 2010-07-16 10:46:27
Size: 4738
Editor: ursinha
Comment: First version
Revision 2 as of 2010-07-16 11:03:23
Size: 5175
Editor: ursinha
Comment: First official draft, for appreciation
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
= QA in the Continuous Rollouts Process = <<TableOfContents>>
Line 3: Line 3:
The proposed merge workflow is in https://dev.launchpad.net/MergeWorkflowDraft. This merge workflow assumes that QA is the blocker of a branch to be rolled out to production, so we need to do the bookkeeping to provide the QA information. = Continuous Rollouts Process: the QA side =

The proposed merge workflow is in https://dev.launchpad.net/MergeWorkflowDraft. This workflow relies on QA, enough to assume a branch able (or not) to be rolled out to production based on its QA state.

We need to do the bookkeeping to provide the QA information to the merger. Details on how we intent to do this are roughly described in this wiki page.
Line 8: Line 12:
A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. But, it's not required to have a bug for each branch. Inform that a branch is [no-qa] when it's the case, and the revision is QA-covered. A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. And, it's NOT required to have a bug for each branch. Informing that a branch is unQAable when it's the case will QA-cover it.
Line 11: Line 15:
== Branches from a QA perspective == == Talking about branches and QA ==
Line 15: Line 19:
== Marking branches appropriately when landing them == == How to make branches QA aware when landing them ==
Line 21: Line 25:

 1.
We can QA it, and it fixes one or more bugs.
=== We can QA the branch, and it fixes one or more bugs ===
Line 27: Line 30:
 2. We can QA it, and it is an incremental step towards one or more bugs. === We can QA the branch, and it is an incremental step towards one or more bugs ===
Line 32: Line 35:
 3. We cannot QA it, and it is an incremental step towards one or more bugs. === We cannot QA the branch, and it is an incremental step towards one or more bugs ===
Line 37: Line 40:
 '''NOTE: don't say that the last branch in an incremental chain is [incr]. Just submit it like category 1.''' '''NOTE: don't say that the last branch in an incremental chain is [incr]. Just submit it like category 1.'''
Line 39: Line 42:
 4. We cannot QA it, but it fixes one or more bugs. === We cannot QA the branch, but it fixes one or more bugs ===
Line 44: Line 47:
 5. We cannot QA it, and it does not have an associated bug. === We cannot QA it, and it does not have an associated bug ===
Line 49: Line 52:
 6. We cannot QA it, and it's just a rollback of a previous bad or unQAed branch. === We cannot QA it, and it's just a rollback of a previous bad or unQAed branch ===
Line 54: Line 57:
Note: Regular branches that fix qa-bad bugs should be submitted like category 1. '''NOTE: Regular branches that fix qa-bad bugs should be submitted like category 1.'''
Line 60: Line 63:
== What the new blesser should do? == == Marking revisions as blessed: the blesser ==
Line 64: Line 67:
The blesser will check each revision for a bug, and: There will be a small application we call "The blesser". It will check each revision for a bug, and mark them as follows:
Line 66: Line 69:
1) If there isn't a bug, it assumes the revision as qa-untestable, revision is blessed.
2) If there's a bug and it's qa-untestable, qa-ok or qa-rollback, revision is blessed.
3) If there's a bug and it's qa-needstesting or qa-bad, revision is not blessed.
 1. If there isn't a bug, it assumes the revision as qa-untestable, revision is blessed.
 2. If there's a bug and it's qa-untestable, qa-ok or qa-rollback, revision is blessed.
 3. If there's a bug and it's qa-needstesting or qa-bad, revision is not blessed.

Here is a diagram that shows how developers and the tagger script will interact with bugs, leaving them ready for the blesser:

{{attachment:bug_machine_state.png}}

Continuous Rollouts Process: the QA side

The proposed merge workflow is in https://dev.launchpad.net/MergeWorkflowDraft. This workflow relies on QA, enough to assume a branch able (or not) to be rolled out to production based on its QA state.

We need to do the bookkeeping to provide the QA information to the merger. Details on how we intent to do this are roughly described in this wiki page.

First, a definition

A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. And, it's NOT required to have a bug for each branch. Informing that a branch is unQAable when it's the case will QA-cover it.

Talking about branches and QA

As a rule, if a branch needs QA, it should be represented by at least one bug. Rarely, QA is not necessary for some branches, or can't be done. Also rarely, some branches cannot be QA'd until more branches land -- they are incremental branches towards a larger bug. Equally rarely are expected branches that only revert others that are bad fixes or rollout blockers: they rollback other branches.

How to make branches QA aware when landing them

To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback]. Note that the idea is to gather as much as possible information in the merge proposal, so you only have to pass parameters to ec2 land and bzr lp-land instead of handcrafting the commit message for it to be in a QA-consistent state.

Each category below describes what you need to do to make the assertion. It also tells you what automation it will trigger. To get your branch landed, you must assert that your branch falls in one of the following categories:

We can QA the branch, and it fixes one or more bugs

  • What you do: Link the bug to the branch in Launchpad.
  • What the tagger does: Each related bug is marked Fix Committed and tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in.

We can QA the branch, and it is an incremental step towards one or more bugs

  • What you do: Link the bug to the branch in Launchpad, and pass --incr or --incremental to ec2 land or bzr lp-land.

  • What the tagger does: Each related bug is tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.

We cannot QA the branch, and it is an incremental step towards one or more bugs

  • What you do: Link the bug to the branch in Launchpad, and pass --incr or --incremental and --no-qa to ec2 land or bzr lp-land.

  • What the tagger does: Each related bug is tagged qa-untestable, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.

NOTE: don't say that the last branch in an incremental chain is [incr]. Just submit it like category 1.

We cannot QA the branch, but it fixes one or more bugs

  • What you do: Link the bug and the branch in Launchpad, and pass --no-qa to ec2 land or bzr lp-land.

  • What the tagger does: Each related bug is marked Fix Committed and tagged qa-untestable.

We cannot QA it, and it does not have an associated bug

  • What you do: Pass --no-qa to ec2 land or bzr lp-land.

  • What the tagger does: Nothing. We include it in a QA report for the current cycle (just a wiki page for now).

We cannot QA it, and it's just a rollback of a previous bad or unQAed branch

  • What you do: Pass --rollback to ec2 land or bzr lp-land.

  • What the tagger does: Each related bug is tagged qa-rollback. The status of the bug will remain the same.

NOTE: Regular branches that fix qa-bad bugs should be submitted like category 1.

PQM will enforce these rules with the usual regex dances. ec2 land and bzr lp-land help you conform to the PQM regex with quicker and more helpful error messages. Alternatively, you can add "[incr]", "[no-qa]" or "[rollback]" in your PQM submission message yourself, if using pqm-submit directly, but we don't recommend it.

Marking revisions as blessed: the blesser

The very important part of having the bugs tagged correctly is that this will determine the QA state of the branch that fixes it, and as a consequence say if the revision is ok to be rolled out to production or not.

There will be a small application we call "The blesser". It will check each revision for a bug, and mark them as follows:

  1. If there isn't a bug, it assumes the revision as qa-untestable, revision is blessed.
  2. If there's a bug and it's qa-untestable, qa-ok or qa-rollback, revision is blessed.
  3. If there's a bug and it's qa-needstesting or qa-bad, revision is not blessed.

Here is a diagram that shows how developers and the tagger script will interact with bugs, leaving them ready for the blesser:

bug_machine_state.png

QAProcessContinuousRollouts (last edited 2010-11-08 10:39:06 by henninge)