Spurious Failures

Occasionally our test suite grows spurious failures, defined as tests that fail in PQM but not on local developer boxes, or tests that fail in unreproducible ways. Such failures break our quality assurance tests and cause PQM to enter testfix mode, where no new code can be landed.

Diagnosing spurious failures

Please improve these rules.

• If the test fails on PQM...
• Run the test on the failing branch locally
• If it fails locally, it's probably not a spurious failure, but a real failure

Real failures

We shouldn't disable failing tests unless they are transient. You should revert the merge.

  bzr merge -r revno..revno_minus_one
  bzr commit -m 'Revert revision #... because of xxx test failing.'
  bzr pqm-submit -m '[rs=flacoste][ui=none] Revert revision #... because of XXX test failing.'

Disable and move on

When a spurious failure is identified, PQM will be in testfix mode. You should disable the test by creating a branch that:

• For doctests, renames doctest.txt to doctest.txt.disabled (verify that the file name does not have a special setup)

• For unittests, renames the test to disabled_test_foo()

Land the branch r=<someone> if you can get the branch reviewed or verified by someone on irc or email. If not, land the branch rs=barry or rs=kiko.

Send an email to the launchpad mailing list identifying the spurious failure that you disabled. Open a bug report and add it to the list below.

Long-term fixes

Figure out why the spurious failure is happening and fix the bug.

List of spurious failures

• https://bugs.edge.launchpad.net/launchpad-project/+bugs?field.tag=spurious-test-failure

See also

https://launchpad.canonical.com/PQMRejectionTroubleshooting