launchpad development
Lesson Learned/Re-learned about Test Isolation
The effort to parallelize our test suite has uncovered many tests that fail due to interaction with other tests. Test isolation is the goal of making sure a given test does not influence the run of another test, regardless of the order they are run. Parallelization exposes the isolation failures because all of our tests are chopped up and spread across different execution units to be run so the set of tests and the order they run in is different from how they are run in our original, monolithic test runs.
Randomizing test runs reveal isolation bugs earlier
If test runs are always randomized (i.e., the --shuffle option is used), then test isolation bugs will become apparent earlier and therefore be easier to fix.
Implicitly relying on ordering of db queries
When querying the database directly or via Storm, the order of the results cannot be assumed unless the query explicitly does an order_by. Usually the results are ordered by row id in our environment but that can be influenced by other database operations. If you expect the results to be in a given order, either sort them in the test or be damned sure the query does the ordering.
Note, it isn't appropriate to change the model to stick in the ordering unless it is really needed elsewhere in the system as needless sorting is expensive.
See 987903 for an example.
Mutable class data
If a class has mutable class data (as opposed to instance data) and it is used in different tests, subsequent tests will get the state of the data as it was modified by previous ones. So if TestA and TestB are run in that order but TestB mucks with mutable class data in a shared class, then no harm is seen. However if the order is reversed, TestA will fail as the state is different from what was expected.
class MockThing:
data = ['initial state']
def addItem(self, x):
self.data.append(x)This MockThing is an example of using mutable class data. It should be written instead as:
class MockThing:
def __init__(self):
self.data = ['initial state']
def addItem(self, x):
self.data.append(x)See 987898.
