Faster Tests

Problem: Our test suite is too slow.

We will know we are done when it takes less than half an hour to land an approved change to a known-working branch.

This page describes a number of solutions which are not at all mutually exclusive.

Parallelizing the test suite

Across machines

Now that we use ec2 for testing, we have access to a virtually unlimited number of machines.

ec2test.py could be changed to optionally spread the tets across multiple ec2 instances. Experiments performed by Robert Collins & Jonathan Lange indicate that the total run time of the suite colud be reduced to 45 minutes or less, with the bulk of this being the time it takes to set up the instances.

Once this work were done, we could use parallelized testing across multiple ec2 instances in our buildbots, or possibly even go back to pre-merge testing.

Across cores

Currently, it's not possible to run more than one instance of the test suite on one machine.

We should fix this so that multiple test suites can be run on the same machine. Once this is done, we can use the very mechanism we use for testing across machines to take advantage of multiple cores when running tests.

Splitting the test suite

By affected code

It's possible to instrument code using coverage to determine what code executes. An appropriate index can then determine what tests are invalidated by a given code delta. --RobertCollins

Gavin hacked on this once: https://launchpad.canonical.com/TestsFromChange --CurtisHovey

By component

Launchpad is made up of multiple components. A change to Bugs will probably not affect the codehosting system.

As we embrace the new tree structure, it will become easier to determine which changes affect which components. If the tree structure accurately reflected the dependencies between components then we could run, say, just the codehosting tests when only codehosting is changed.

This would reduce the number of tests that needed to be run before getting known-good. For extra safety, we could run the full test suite post-merge as part of a continuous integration environment.

By test type

We could split the test suite so that there are unit tests and integration tests. Here, 'unit test' does not necessarily mean a test without layers (as in Zope terminology) or a test that subclasses unittest.TestCase but rather a test that aims to test exactly one thing. 'Integration test' means a test that combines multiple elements of the system and tests their behaviour end-to-end. In general, unit tests are much faster than integration tests, since they exercise less of the system.

If we split along these lines, then the unit tests would be run pre-merge, and the integration tests would be run post-merge as part of a continuous integration system.

Making the tests faster

Our tests, even our unit tests, are too slow. Here are some ways we can make them faster.

Use simpler layers

Many tests use LaunchpadFunctionalLayer when they don't need the librarian. These tests should be changed to use DatabaseFunctionalLayer.

Many tests use on the PageTestLayer when the DatabaseFunctionalLayer or even lower is needed.

Use testresources

Switching to testresources would mean that each test would ask for what it needed, rather than selecting some bulky atomic layer. This would prevent unnecessary work.

Get rid of sampledata

The Landscape team claim that our use of sample data is slowing down our database-using tests. We should experimentally verify this claim.

• We could consider an EmptyDatabaseLayer that makes no attempt to load sampledata. --Julian

Commit less

Many LaunchpadObjectFactory methods call commit() repeatedly. This almost certainly makes the tests that use them much slower than they need to be.

Running in ramdisk

It's possible that running the whole suite in a ramdisk (not just postgres) could provide a performance benefit. We should experimentally verify this.

Test doubles

Using a postgresql database is always going to be slower than manipulating objects in memory. If we had a reliable set of test doubles (mocks, fakes, whatever), we could use those for many of our tests.

Speeding up the build process

'make build' takes about 30 minutes on an empty tree. This is pretty terrible. Using a binary distribution mechanism rather than the source based one we currently use would help a lot here.

Separate model objects from the database

Much of the behaviour of our core model objects is not actually tied to the database. We should find some way to get at, say, a Bug or a Branch object without doing database queries.

Mock database

The vast bulk of our tests issue database queries in a deterministic fashion. It is possible to record the results of a test so subsequent runs of the same test will replay the results from the record rather than by actually hitting the database. The proof of concept was ready but was victimized by the Storm switch, but the bulk of the work is already in the tree.

Reducing the number of tests

Many of our tests test the same thing more than once. This is partly due to unclear layering in our architecture and partly due to lack of good test doubles. However, there are almost certainly many tests that are simply duplicating the work of others.

We can find this duplicated work by having clearer mappings from code modules to test locations. If we can find tests predictably, then we will at least have an aid for preventing further duplication.

Code coverage tools can also help us identify test duplication.