Spurious Failures

Occasionally our test suite grows spurious failures, defined as tests that fail in PQM but not on local developer boxes, or tests that fail in unreproducible ways. Such failures break our quality assurance tests and cause PQM to enter testfix mode, where no new code can be landed.

Diagnosing spurious failures

Please improve these rules.

• If the test fails on PQM...
• Run the test on the failing branch locally
• If it fails locally, it's probably not a spurious failure, but a real failure

Disable and move on

When a spurious failure is identified, PQM will be in testfix mode. You should disable the test by creating a branch that:

• For doctests, renames doctest.txt to doctest.txt.disabled

• For unittests, renames the test to disabled_test_foo()

Land the branch r=<someone> if you can get the branch reviewed or verified by someone on irc or email. If not, land the branch rs=barry or rs=kiko.

Send an email to the launchpad mailing list identifying the spurious failure that you disabled. Open a bug report and add it to the list below.

Long-term fixes

Figure out why the spurious failure is happening and fix the bug.

List of spurious failures

• Bug XXXXXX: Race condition in frobnication

See also

https://launchpad.canonical.com/PQMRejectionTroubleshooting