Change Launchpad, Find Tests

jsk had a dream of a day when a diff could be used to find the exact set of tests to run, thus giving him mucho confidence before submitting a branch for review, but without the long wait to run make check, and without needing a Tillenius-like understanding of the codebase and her mysterious ways. And so it came to pass...

jsk and I did some early work on such a tool at AllHands (2007). I did some more work on it after that, and now have a prototype:

bzr+ssh://devpad.canonical.com/code/gavin.panella/launchpad/zope-coverage-per-test

Uses

1. Navigating the code base.
   • For someone new to the code this will help them discover the tests relevant to their area.
   • Our tests are also documentation, so this index also helps us find relevant documentation.

2. Increasing velocity. Because we can quickly identify the tests we need to run, we can:

   • avoid a lengthy test run before submitting code for review,
   • reduce the likelihood of unforeseen issues in PQM when we don't have time for a full test run beforehand,
   • and thus reduce the likelihood of subsequent re-review.

Getting started

There are two parts to this tool:

1. Recording:

   • A patch to the Zope tree to let us record coverage information for individual tests.
   • A tool to create an index from that coverage information.

2. Lookup:

   • A tool that uses the index to find tests relevant to changes in a development branch of Launchpad.

If you just want to see how this might be useful to you as a developer, skip to step 2.

Recording

The branch has an example set-up script, utilities/tfc-setup.sh (also here). This will create you a new LP branch with a customised sourcecode/zope.

Once set-up, there are two steps:

1. Run the test suite with coverage options. This means running bin/test with --coverage=<directory> and (new) --coverage-per-test. This creates a dump file containing tests and the source files (and lines) they exercise.

2. Run utilities/tfc.py --index to create an index file which will let us map from source files+lines back to the tests.

Example

Run the test suite, recording coverage.

$ bin/test -vvt lib/canonical/ --coverage=coverage --coverage-per-test
Running tests at level 1
Running unit tests:
...
Total: 10850 tests, 0 failures, 0 errors

Make the index.

$ ./sourcecode/zope/utilities/tfc.py --index coverage/counts.{dump,index}
$ du -h coverage/*
98M     coverage/counts.dump
396K    coverage/counts.index

Lookup

Pull the branch. The tool you need is utilities/tfc.py. You also need the index file from the recording phase.

• Conveniently, I've baked an index already: counts.index. It'll be out of date by the time you try this, but will probably be good enough as an example.

Next you just run utilities/tfc.py --lookup to figure out the list of the tests we need to run. This actually returns a regular expression which can be used as the -t argument to bin/test.

Example

Get a copy of the zope-coverage-per-test branch.

bzr branch bzr+ssh://devpad.canonical.com/code/gavin.panella/launchpad/zope-coverage-per-test

Work in a new rocketfuel branch.

bzr branch rocketfuel tfc-example-branch
cd tfc-example-branch

Make changes to the source tree. For example, add an extra space to line 81 of lib/canonical/widgets/project.py:

sed -i '81s/==/== /' lib/canonical/widgets/project.py

Download the index file.

wget -O counts.index --user=warthogs --password=${password:?} \
     'https://launchpad.canonical.com/TestsFromChanges?action=AttachFile&do=get&target=counts.index'

Now lookup the relevant tests.

bzr diff | ../zope-coverage-per-test/utilities/tfc.py --lookup-and-print counts.index

Or choose another option.

Usage: tfc.py --index <dump file> <index file>
       tfc.py --lookup|--lookup-and-print <index file>

Options:
  -h, --help          show this help message and exit
  --index             Make an index file from the coverage dump.
  --lookup            Accept a unified diff on stdin and return a regular
                      expression that will select the tests that should be
                      run.
  --lookup-and-print  Accept a unified diff on stdin and print out the
                      relevant tests that should be run.
  --debug             Show details of each step.

Notes

Running the full test suite with coverage options would be centralised and run continuously. Even so, it would always lag behind rocketfuel (unless PQM is very quiet), but I don't think that's critical. This tool is meant to be as accurate as possible, but doesn't need to be 100%.

The zope-coverage-per-test branch does not need to be merged into RF, it only needs to be merged into the centralised coverage service. The index file produced is quite small and could easily be downloaded regularly, say daily, to developer machines. Only the helper tool would be needed on developer machines for look-ups.

The look-up process could be provided as a web-service, and then no specific tools would be needed on developer machines. But this does not work if you're disconnected.

Comments

This looks very fun. -- CurtisHovey, 2007-11-22

Hey guys, this looks very cool. Did you think about taking a more static approach and just marking up files to indicate where their tests live? I'm not sure which approach I'd prefer, just thinking aloud. -- JonathanLange, 2007-11-27

• Hi Jonathan. The association it records is from individual lines in source files back to the tests that exercise them, so too fine-grained to mark up the source code, though a coarser source file -> test index would be trivial to derive from that. I like the idea, especially because we would be reminded every time we open a source file of what tests hit it, so giving us an awareness of the codebase. Potentially good for learning and gardening. -- GavinPanella, 2007-11-27.