Differences between revisions 3 and 4

Contents

Continuous Rollouts Process: the QA side

Continuous Rollouts Process: the QA side

The proposed merge workflow is in https://dev.launchpad.net/MergeWorkflowDraft. This workflow relies on QA, enough to assume a branch able (or not) to be rolled out to production based on its QA state.

We need to do the bookkeeping to provide the QA information to the merger. Details on how we intent to do this are described in this wiki page.

Definition of QA

A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. But, it's NOT required to have a bug for each branch. Informing that a branch is unQAable when landing it will QA-cover it.

A fix is considered QAed when falling into one of the following categories:

States of QA

qa-ok: The branch is verified and implements successfully what it proposes.
qa-untestable: The branch cannot be tested separately of the rest of the fix or cannot be tested at all, and is harmless.
qa-bad: The branch is verified and doesn't work as it should or causes regressions.
qa-rollback: The branch reverts a previous one that's in a blocker state (qa-bad or qa-needstesting).
qa-needstesting: The branch has landed and is waiting to be QAed.

qa-untestable, qa-rollback and qa-needstesting should be set by the Tagger. qa-bad and qa-ok should be manually set. The diagram shows the QA states of the bug and what causes them.

Connecting branches and QA

As a rule, if a branch needs QA, it should be represented by at least one bug. Rarely, QA is not necessary for some branches, or can't be done. Also rarely, some branches cannot be QA'd until more branches land -- they are incremental branches towards a larger bug. Equally rarely are expected branches that only revert others that are bad fixes or rollout blockers: they rollback other branches.

How to make branches QA-aware when landing them

To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback]. Note that the idea is to gather as much information as possible in the merge proposal, so the developer only has to use parameters on ec2 land and bzr lp-land instead of handcrafting the commit message for it to be recognized as in a QA-consistent state.

Each category below describes what the developer needs to do to make the assertion. It also tells the developer what automation it will trigger. In order to get the branch landed, the developer must assert that the branch falls in one of the following categories:

We can QA the branch, and it fixes one or more bugs (most common)

Developer action: Link the bug to the branch in Launchpad. (Alternative way).
Tagger action: Each related bug is marked Fix Committed and tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in.

We can QA the branch, and it is an incremental step towards the fix of one or more bugs

Developer action: Link the bug to the branch in Launchpad (Alternative way), and use --incremental on ec2 land or bzr lp-land.
Tagger action: Each related bug is tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.

We cannot QA the branch, and it is an incremental step towards the fix of one or more bugs

Developer action: Link the bug to the branch in Launchpad (Alternative way), and use --incremental and --no-qa on ec2 land or bzr lp-land.
Tagger action: Each related bug is tagged qa-untestable, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.

NOTE: Developer shouldn't use the --incremental option when landing the last branch in an incremental chain. Branch must be submitted like the most common case.

We cannot QA the branch, but it fixes one or more bugs

Developer action: Link the bug and the branch in Launchpad (Alternative way), and use --no-qa on ec2 land or bzr lp-land.
Tagger action: Each related bug is marked Fix Committed and tagged qa-untestable.

We cannot QA it, and it does not have an associated bug

Developer action: Use --no-qa on ec2 land or bzr lp-land.
Tagger action: Nothing. We include it in a QA report for the current cycle (just a wiki page for now).

We cannot QA it, and it's just a rollback of a previous qa-bad or blocker qa-needstesting branch

Developer action: Use --rollback on ec2 land or bzr lp-land.
Tagger action: Each related bug is tagged qa-rollback. The status of the bug will remain the same.

NOTE: Regular branches that fix qa-bad bugs should be submitted like the most common case.

Alternative way to linking bugs to branches using Launchpad UI

Using --fixes when doing local commits add the metadata to the branch. When the branch is pushed to Launchpad, it scans the revisions and automatically links the mentioned bugs to the branch.

PQM will enforce these rules with the usual regex dances. ec2 land and bzr lp-land help the developer conform to the PQM regex with quicker and more helpful error messages. Alternatively, the developer can add "[incr]", "[no-qa]" or "[rollback]" in the PQM submission message, if using pqm-submit directly, but it's not recommended considering it's not needed and other tools with faster response automatically check and enforce the tags.

Marking revisions as blessed: the blesser

The very important part of having the bugs tagged correctly is that this will determine the QA state of the branch that fixes it, and as a consequence say if the revision is blessed to be rolled out to production of if it's blocked, and therefore should not be rolled out.

There will be a small application called "the blesser". It will check each revision landed on the development branch, check the existence of bugs and commit messages, and mark them as follows:

No bug: Blesser assumes the revision is qa-untestable, therefore revision is blessed.
Bug AND it's qa-untestable, qa-ok or qa-rollback: Blesser assumes that's ok, therefore revision is blessed.
Bug AND it's qa-needstesting or qa-bad: Blesser blocks the rollouts by marking the revision as blocked.

Here is a diagram that shows how developers and the tagger script will interact with bugs, leaving them ready for the blesser:

-  ⇤ ← Revision 3 as of 2010-07-16 11:07:00 → 
  Size: 5187
  Editor: ursinha
  Comment:
+   ← Revision 4 as of 2010-07-16 14:04:59 → ⇥
  Size: 6739
  Editor: ursinha
  Comment: Changed according to diogo's suggestions
-Deletions are marked like this.
+Additions are marked like this.
 Line 7:
-We need to do the bookkeeping to provide the QA information to the merger. Details on how we intent to do this are roughly described in this wiki page.
+We need to do the bookkeeping to provide the QA information to the merger. Details on how we intent to do this are described in this wiki page.
 Line 9:
+== Definition of QA ==
-Line 10:
+Line 11:
-== First, a definition ==
+A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. But, it's NOT required to have a bug for each branch. Informing that a branch is unQAable when landing it will QA-cover it.
-Line 12:
+Line 13:
-A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. And, it's NOT required to have a bug for each branch. Informing that a branch is unQAable when it's the case will QA-cover it.
+A fix is considered QAed when falling into one of the following categories:
-Line 14:
+Line 15:
+=== States of QA ===
-Line 15:
+Line 17:
-== Talking about branches and QA ==
+ * qa-ok: The branch is verified and implements successfully what it proposes.
 * qa-untestable: The branch cannot be tested separately of the rest of the fix or cannot be tested at all, and is harmless.
 * qa-bad: The branch is verified and doesn't work as it should or causes regressions.
 * qa-rollback: The branch reverts a previous one that's in a blocker state (qa-bad or qa-needstesting).
 * qa-needstesting: The branch has landed and is waiting to be QAed.

qa-untestable, qa-rollback and qa-needstesting should be set by the Tagger. qa-bad and qa-ok should be manually set. The diagram shows the QA states of the bug and what causes them.

== Connecting branches and QA ==
-Line 19:
+Line 29:
-== How to make branches QA aware when landing them ==
+== How to make branches QA-aware when landing them ==
-Line 21:
+Line 31:
-To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback]. Note that the idea is to gather as much as possible information in the merge proposal, so you only have to pass parameters to {{{ec2 land}}} and {{{bzr lp-land}}} instead of handcrafting the commit message for it to be in a QA-consistent state.
+To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback]. Note that the idea is to gather as much information as possible in the merge proposal, so the developer only has to use parameters on {{{ec2 land}}} and {{{bzr lp-land}}} instead of handcrafting the commit message for it to be recognized as in a QA-consistent state.
-Line 23:
+Line 33:
-Each category below describes what you need to do to make the assertion.  It also tells you what automation it will trigger. To get your branch landed, you must assert that your branch falls in one of the following categories:
+Each category below describes what the developer needs to do to make the assertion.  It also tells the developer what automation it will trigger. In order to get the branch landed, the developer must assert that the branch falls in one of the following categories:
-Line 25:
+Line 35:
-=== We can QA the branch, and it fixes one or more bugs ===
+=== We can QA the branch, and it fixes one or more bugs (most common) ===
-Line 27:
+Line 37:
-  * What you do: Link the bug to the branch in Launchpad.
  * What the tagger does: Each related bug is marked Fix Committed and tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in.
+  * Developer action: Link the bug to the branch in Launchpad. ([[Alternative way]]). 
  * Tagger action: Each related bug is marked Fix Committed and tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in.
-Line 30:
+Line 40:
-=== We can QA the branch, and it is an incremental step towards one or more bugs ===
+=== We can QA the branch, and it is an incremental step towards the fix of one or more bugs ===
-Line 32:
+Line 42:
-  * What you do: Link the bug to the branch in Launchpad, and pass {{{--incr}}} or {{{--incremental}}} to {{{ec2 land}}} or {{{bzr lp-land}}}.
  * What the tagger does: Each related bug is tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.
+  * Developer action: Link the bug to the branch in Launchpad ([[Alternative way]]), and use {{{--incremental}}} on {{{ec2 land}}} or {{{bzr lp-land}}}.
  * Tagger action: Each related bug is tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.
-Line 35:
+Line 45:
-=== We cannot QA the branch, and it is an incremental step towards one or more bugs ===
+=== We cannot QA the branch, and it is an incremental step towards the fix of one or more bugs ===
-Line 37:
+Line 47:
-  * What you do: Link the bug to the branch in Launchpad, and pass {{{--incr}}} or {{{--incremental}}} and {{{--no-qa}}} to {{{ec2 land}}} or {{{bzr lp-land}}}.
  * What the tagger does: Each related bug is tagged qa-untestable, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.
+  * Developer action: Link the bug to the branch in Launchpad ([[Alternative way]]), and use {{{--incremental}}} and {{{--no-qa}}} on {{{ec2 land}}} or {{{bzr lp-land}}}.
  * Tagger action: Each related bug is tagged qa-untestable, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.
-Line 40:
+Line 50:
-'''NOTE: don't say that the last branch in an incremental chain is [incr].  Just submit it like category 1.'''
+'''NOTE: Developer shouldn't use the {{{--incremental}}} option when landing the last branch in an incremental chain. Branch must be submitted like the most common case.'''
-Line 44:
+Line 54:
-  * What you do: Link the bug and the branch in Launchpad, and pass {{{--no-qa}}} to {{{ec2 land}}} or {{{bzr lp-land}}}.
  * What the tagger does: Each related bug is marked Fix Committed and tagged qa-untestable.
+  * Developer action: Link the bug and the branch in Launchpad ([[Alternative way]]), and use {{{--no-qa}}} on {{{ec2 land}}} or {{{bzr lp-land}}}.
  * Tagger action: Each related bug is marked Fix Committed and tagged qa-untestable.
-Line 49:
+Line 59:
-  * What you do: Pass {{{--no-qa}}} to {{{ec2 land}}} or {{{bzr lp-land}}}.
  * What the tagger does: Nothing. We include it in a QA report for the current cycle (just a wiki page for now).
+  * Developer action: Use {{{--no-qa}}} on {{{ec2 land}}} or {{{bzr lp-land}}}.
  * Tagger action: Nothing. We include it in a QA report for the current cycle (just a wiki page for now).
-Line 52:
+Line 62:
-=== We cannot QA it, and it's just a rollback of a previous bad or unQAed branch ===
+=== We cannot QA it, and it's just a rollback of a previous qa-bad or blocker qa-needstesting branch ===
-Line 54:
+Line 64:
-  * What you do: Pass {{{--rollback}}} to {{{ec2 land}}} or {{{bzr lp-land}}}.
  * What the tagger does: Each related bug is tagged qa-rollback. The status of the bug will remain the same.
+  * Developer action: Use {{{--rollback}}} on {{{ec2 land}}} or {{{bzr lp-land}}}.
  * Tagger action: Each related bug is tagged qa-rollback. The status of the bug will remain the same.
-Line 57:
+Line 67:
-'''NOTE: Regular branches that fix qa-bad bugs should be submitted like category 1.'''
+'''NOTE: Regular branches that fix qa-bad bugs should be submitted like the most common case.'''

=== Alternative way to linking bugs to branches using Launchpad UI ===

Using --fixes when doing local commits add the metadata to the branch. When the branch is pushed to Launchpad, it scans the revisions and automatically links the mentioned bugs to the branch.
-Line 60:
+Line 74:
-PQM will enforce these rules with the usual regex dances. {{{ec2 land}}} and {{{bzr lp-land}}} help you conform to the PQM regex with quicker and more helpful error messages.  Alternatively, you can add "[incr]", "[no-qa]" or "[rollback]" in your PQM submission message yourself, if using pqm-submit directly, but we don't recommend it.
+PQM will enforce these rules with the usual regex dances. {{{ec2 land}}} and {{{bzr lp-land}}} help the developer conform to the PQM regex with quicker and more helpful error messages.  Alternatively, the developer can add "[incr]", "[no-qa]" or "[rollback]" in the PQM submission message, if using pqm-submit directly, but it's not recommended considering it's not needed and other tools with faster response automatically check and enforce the tags.
-Line 65:
+Line 79:
-The very important part of having the bugs tagged correctly is that this will determine the QA state of the branch that fixes it, and as a consequence say if the revision is ok to be rolled out to production or not.
+The very important part of having the bugs tagged correctly is that this will determine the QA state of the branch that fixes it, and as a consequence say if the revision is '''''blessed''''' to be rolled out to production of if it's '''''blocked''''', and therefore should not be rolled out.
-Line 67:
+Line 81:
-There will be a small application we call "The blesser". It will check each revision for a bug, and mark them as follows:
+There will be a small application called "the blesser". It will check each revision landed on the development branch, check the existence of bugs and commit messages, and mark them as follows:
-Line 69:
+Line 83:
-. If there isn't a bug, it assumes the revision as qa-untestable, revision is blessed. 
 2. If there's a bug and it's qa-untestable, qa-ok or qa-rollback, revision is blessed.
 3. If there's a bug and it's qa-needstesting or qa-bad, revision is not blessed.
+. No bug: Blesser assumes the revision is qa-untestable, therefore revision is blessed. 
 2. Bug AND it's qa-untestable, qa-ok or qa-rollback: Blesser assumes that's ok, therefore revision is blessed.
 3. Bug AND it's qa-needstesting or qa-bad: Blesser blocks the rollouts by marking the revision as blocked.

launchpad development

Diff for "QAProcessContinuousRollouts"