SpuriousFailures

Not logged in - Log In / Register

Spurious Failures

Occasionally our test suite grows spurious failures, defined as tests that fail in PQM but not on local developer boxes, or tests that fail in unreproducible ways. Such failures break our quality assurance tests and cause PQM to enter testfix mode, where no new code can be landed.

Diagnosing spurious failures

Please improve these rules.

Real failures

We shouldn't disable failing tests unless they are transient. You should revert the merge.

  bzr merge -r revno..revno_minus_one
  bzr commit -m 'Revert revision #... because of xxx test failing.'
  bzr pqm-submit -m '[rs=flacoste][ui=none] Revert revision #... because of XXX test failing.'

Disable and move on

When a spurious failure is identified, PQM will be in testfix mode. You should disable the test by creating a branch that:

Land the branch r=<someone> if you can get the branch reviewed or verified by someone on irc or email. If not, land the branch r=<you>.

Send an email to the launchpad mailing list identifying the spurious failure that you disabled. Open a bug report and add it to the list below.

Long-term fixes

Figure out why the spurious failure is happening and fix the bug.

List of spurious failures

See also

https://launchpad.canonical.com/PQMRejectionTroubleshooting

SpuriousFailures (last edited 2011-10-28 20:35:52 by allenap)