Diff for "QAProcessContinuousRollouts"

Not logged in - Log In / Register
Differences between revisions 11 and 12
⇤ ← Revision 11 as of 2010-07-22 21:45:50 →
Size: 8958
Editor: mars
Comment:
← Revision 12 as of 2010-08-18 00:40:00 → ⇥
Size: 9158
Editor: lifeless
Comment: updated workflow to handle bugs having many branches (candidate fix that is broken, then rollback, then real fix)
Deletions are marked like this. Additions are marked like this.
Line 19: Line 19:
 * qa-bad: The branch is verified and doesn't work as it should or causes regressions.
 * qa-rollback: The branch reverts a previous one that's in a blocker state (qa-bad or qa-needstesting).		  * qa-bad: The branch is verified and doesn't work as it should or causes regressions. This is derived by having a bad-commit with no rollback=XXX for it in devel. bad-commit tags are of the form bad-commit-REVNOINDEVEL.
Line 23: Line 22:
qa-untestable, qa-rollback and qa-needstesting should be set by the [[http://launchpad.net/qa-tagger|tagger]]. qa-bad and qa-ok should be manually set by the developer. The diagram at the end of this page shows the QA states of the bug and the changes that trigger them. qa-untestable, and qa-needstesting should be set by the [[http://launchpad.net/qa-tagger|tagger]]. qa-bad and qa-ok should be manually set by the developer. The diagram at the end of this page shows the QA states of the bug and the changes that trigger them.
Line 31: Line 30:
To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback]. Note that the idea is to gather as much information as possible in the merge proposal, so the developer only has to use parameters on {{{ec2 land}}} and {{{bzr lp-land}}} instead of handcrafting the commit message for it to be recognized as in a QA-consistent state. To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback=xxx]. Note that the idea is to gather as much information as possible in the merge proposal, so the developer only has to use parameters on {{{ec2 land}}} and {{{bzr lp-land}}} instead of handcrafting the commit message for it to be recognized as in a QA-consistent state.
Line 87: Line 86:
      $ ./ec2 land --rollback https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666       $ ./ec2 land --rollback=BROKENBRANCHREVNO https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666
Line 89: Line 88:
  * '''Tagger action''': Each related bug is tagged qa-rollback. The status of the bug will remain the same.   * '''Tagger action''': Each bug has the qa-bad tag removed putting it back to no qa tags, and the bad-commit-* tag left in place. The status of the bug will remain the same.
Line 94: Line 93:
PQM will enforce these rules with the usual regex dances. {{{ec2 land}}} and {{{bzr lp-land}}} help the developer conform to the PQM regex with quicker and more helpful error messages. Alternatively, the developer can add "[incr]", "[no-qa]" or "[rollback]" in the PQM submission message, if using pqm-submit directly, but it's not recommended considering it's not needed and other tools with faster response automatically check and enforce the tags. PQM will enforce these rules with the usual regex dances. {{{ec2 land}}} and {{{bzr lp-land}}} help the developer conform to the PQM regex with quicker and more helpful error messages. Alternatively, the developer can add "[incr]", "[no-qa]" or "[rollback=xxxxx]" in the PQM submission message, if using pqm-submit directly, but it's not recommended considering it's not needed and other tools with faster response automatically check and enforce the tags.
Line 105: Line 104:
 3. '''Has bug''' and it's '''qa-needstesting''' or '''qa-bad''': Blesser blocks the rollouts by marking the revision as '''blocked'''.  3. '''Has bug''' and it's '''qa-needstesting'''
 4. '''Has bug''' tagged '''qa-bad''': Blesser blocks the rollouts by marking the revision as '''blocked''' unless a later revision has rollback=xxx where xxx is this revision.
Line 107: Line 107:

Contents

  1. Continuous Rollouts Process: the QA side
    1. Definition of QA
      1. States of QA
    2. Connecting branches and QA
    3. How to make branches QA-aware when landing them
      1. We can QA the branch, and it fixes one or more bugs (most common)
      2. We can QA the branch, and it is an incremental step towards the fix of one or more bugs
      3. We cannot QA the branch, and it is an incremental step towards the fix of one or more bugs
      4. We cannot QA the branch, but it fixes one or more bugs
      5. We cannot QA it, and it does not have an associated bug
      6. We cannot QA it, and it's just a rollback of a previous qa-bad or blocker qa-needstesting branch
    4. Marking revisions as blessed: the blesser
      1. The Blesser Implementation

Continuous Rollouts Process: the QA side

The proposed merge workflow is in https://dev.launchpad.net/MergeWorkflowDraft. This workflow relies on QA, enough to assume a branch able (or not) to be rolled out to production based on its QA state.

We need to do the bookkeeping to provide the QA information to the mechanism that will do the rollouts. Details on how we intent to do this are described in this wiki page.

Definition of QA

A bug is the smallest unit of QA. So, deciding if something needs QA is based on the existence of the bug. But, it's NOT required to have a bug for each branch. Informing that a branch is unQAable when landing it will QA-cover it.

A fix is considered QAed when falling into one of the following categories:

States of QA

  • qa-ok: The branch is verified and implements successfully what it proposes.
  • qa-untestable: The branch cannot be tested separately of the rest of the fix or cannot be tested at all, and is harmless.
  • qa-bad: The branch is verified and doesn't work as it should or causes regressions. This is derived by having a bad-commit with no rollback=XXX for it in devel. bad-commit tags are of the form bad-commit-REVNOINDEVEL.
  • qa-needstesting: The branch has landed and is waiting to be QAed.

qa-untestable, and qa-needstesting should be set by the tagger. qa-bad and qa-ok should be manually set by the developer. The diagram at the end of this page shows the QA states of the bug and the changes that trigger them.

Connecting branches and QA

As a rule, if a branch needs QA, it should be represented by at least one bug. Rarely, QA is not necessary for some branches, or can't be done. Also rarely, some branches cannot be QA'd until more branches land -- they are incremental branches towards a larger bug. Equally rarely are expected branches that only revert others that are bad fixes or rollout blockers: they rollback other branches.

How to make branches QA-aware when landing them

To represent and sign those situations, there are three new commit message tags: [no-qa], [incr] and [rollback=xxx]. Note that the idea is to gather as much information as possible in the merge proposal, so the developer only has to use parameters on ec2 land and bzr lp-land instead of handcrafting the commit message for it to be recognized as in a QA-consistent state.

The developer should set the commit message in the merge proposal, without worrying about the QA clauses (incremental, bug, no-qa and rollback), and the helper scripts should generate the correct PQM commit message.

Each category below describes what the developer needs to do to make the assertion. It also tells the developer what automation it will trigger. For simplicity we're using ec2 land in all the examples. In order to get the branch landed, the developer must assert that the branch falls in one of the following categories:

We can QA the branch, and it fixes one or more bugs (most common)

  • Developer action: Link the bug to the branch in Launchpad (see alternative below), and use ec2 land or bzr lp-land: 

      $ ./ec2 land https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666

  • Tagger action: Each related bug is marked Fix Committed and tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in.

TIP

Alternative way to link bugs to branches in Launchpad, automatically: Instead of accessing the branch page in Launchpad and linking the bugs manually after pushing it, the developer can use bzr commit --fixes=123456. It adds the information of the bug fixed in the branch metadata. When the same branch is pushed to Launchpad, it's scanned and provided fixed bugs are automatically linked to the branch in Launchpad.

We can QA the branch, and it is an incremental step towards the fix of one or more bugs

  • Developer action: Link the bug to the branch in Launchpad (see alternative above), and use --incremental on ec2 land or bzr lp-land. 

      $ ./ec2 land --incremental https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666

  • Tagger action: Each related bug is tagged qa-needstesting, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.

We cannot QA the branch, and it is an incremental step towards the fix of one or more bugs

  • Developer action: Link the bug to the branch in Launchpad (see alternative above), and use --incremental and --no-qa on ec2 land or bzr lp-land. 

      $ ./ec2 land --incremental --no-qa https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666

  • Tagger action: Each related bug is tagged qa-untestable, and a comment is added mentioning the revision number and branch name it landed in. The status of the bug will remain In Progress.

NOTE:

Developers shouldn't use the --incremental option when landing the last branch in an incremental chain. Branch should be submitted like the most common case.

We cannot QA the branch, but it fixes one or more bugs

  • Developer action: Link the bug and the branch in Launchpad (see alternative above), and use --no-qa on ec2 land or bzr lp-land. 

      $ ./ec2 land --no-qa https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666

  • Tagger action: Each related bug is marked Fix Committed and tagged qa-untestable.

We cannot QA it, and it does not have an associated bug

  • Developer action: Use --no-qa on ec2 land or bzr lp-land. 

      $ ./ec2 land --no-qa https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666

  • Tagger action: Nothing. We include it in a QA report for the current cycle (just a wiki page for now).

We cannot QA it, and it's just a rollback of a previous qa-bad or blocker qa-needstesting branch

  • Developer action: Use --rollback on ec2 land or bzr lp-land. 

      $ ./ec2 land --rollback=BROKENBRANCHREVNO https://code.launchpad.net/~developer/launchpad/the-branch-bug-1234/+merge/66666

  • Tagger action: Each bug has the qa-bad tag removed putting it back to no qa tags, and the bad-commit-* tag left in place. The status of the bug will remain the same.

NOTE:

Regular branches that fix qa-bad bugs should be submitted like the most common case.

PQM will enforce these rules with the usual regex dances. ec2 land and bzr lp-land help the developer conform to the PQM regex with quicker and more helpful error messages. Alternatively, the developer can add "[incr]", "[no-qa]" or "[rollback=xxxxx]" in the PQM submission message, if using pqm-submit directly, but it's not recommended considering it's not needed and other tools with faster response automatically check and enforce the tags.

Marking revisions as blessed: the blesser

The very important part of having the bugs tagged correctly is that this will determine the QA state of the branch that fixes it, and as a consequence say if the revision is blessed to be rolled out to production of if it's blocked, and therefore should not be rolled out.

A small application, "the blesser", will check each revision landed on the development branch, check the existence of bugs and commit messages, and mark them as follows:

  1. Has no bug: Blesser assumes the revision is qa-untestable, therefore revision is blessed.

  2. Has bug and it's qa-untestable, qa-ok or qa-rollback: Blesser assumes that's ok, therefore revision is blessed.

  3. Has bug and it's qa-needstesting

  4. Has bug tagged qa-bad: Blesser blocks the rollouts by marking the revision as blocked unless a later revision has rollback=xxx where xxx is this revision.

  5. Has bug and has no QA tags (e.g. removed manually): Blesser blocks the rollouts by marking the revision as blocked.

Here is a diagram that shows how developers and the tagger script will interact with bugs, leaving them ready for the blesser:

bug_machine_statev2.png

The Blesser Implementation

The dev notes for the blesser tool are on the QAShepherd page.

QAProcessContinuousRollouts (last edited 2010-11-08 10:39:06 by henninge)