launchpad developmentThis page is outdated, and is here for reference only. |
Change Launchpad, Find Tests
jsk had a dream of a day when a diff could be used to find the exact set of tests to run, thus giving him mucho confidence before submitting a branch for review, but without the long wait to run make check, and without needing a Tillenius-like understanding of the codebase and her mysterious ways. And so it came to pass...
jsk and I did some early work on such a tool at AllHands (2007). I did some more work on it after that, and now have a prototype:
bzr+ssh://devpad.canonical.com/code/gavin.panella/launchpad/zope-coverage-per-test
Uses
- Navigating the code base.
- For someone new to the code this will help them discover the tests relevant to their area.
- Our tests are also documentation, so this index also helps us find relevant documentation.
Increasing velocity. Because we can quickly identify the tests we need to run, we can:
- avoid a lengthy test run before submitting code for review,
- reduce the likelihood of unforeseen issues in PQM when we don't have time for a full test run beforehand,
- and thus reduce the likelihood of subsequent re-review.
Getting started
There are two parts to this tool:
- A patch to the Zope tree to let us record coverage information for individual tests.
- A tool to create an index from that coverage information.
- A tool that uses the index to find tests relevant to changes in a development branch of Launchpad.
If you just want to see how this might be useful to you as a developer, skip to step 2.
Recording
The branch has an example set-up script, utilities/tfc-setup.sh (also here). This will create you a new LP branch with a customised sourcecode/zope.
Once set-up, there are two steps:
Run the test suite with coverage options. This means running bin/test with --coverage=<directory> and (new) --coverage-per-test. This creates a dump file containing tests and the source files (and lines) they exercise.
Run utilities/tfc.py --index to create an index file which will let us map from source files+lines back to the tests.
Example
Run the test suite, recording coverage.
$ bin/test -vvt lib/canonical/ --coverage=coverage --coverage-per-test Running tests at level 1 Running unit tests: ... Total: 10850 tests, 0 failures, 0 errors
Make the index.
$ ./sourcecode/zope/utilities/tfc.py --index coverage/counts.{dump,index}
$ du -h coverage/*
98M coverage/counts.dump
396K coverage/counts.index
Lookup
Pull the branch. The tool you need is utilities/tfc.py. You also need the index file from the recording phase.
Conveniently, I've baked an index already: counts.index. It'll be out of date by the time you try this, but will probably be good enough as an example.
Next you just run utilities/tfc.py --lookup to figure out the list of the tests we need to run. This actually returns a regular expression which can be used as the -t argument to bin/test.
Example
Get a copy of the zope-coverage-per-test branch.
bzr branch bzr+ssh://devpad.canonical.com/code/gavin.panella/launchpad/zope-coverage-per-test
Work in a new rocketfuel branch.
bzr branch rocketfuel tfc-example-branch cd tfc-example-branch
Make changes to the source tree. For example, add an extra space to line 81 of lib/canonical/widgets/project.py:
sed -i '81s/==/== /' lib/canonical/widgets/project.py
Download the index file.
wget -O counts.index --user=warthogs --password=${password:?} \
'https://launchpad.canonical.com/TestsFromChanges?action=AttachFile&do=get&target=counts.index'Now lookup the relevant tests.
bzr diff | ../zope-coverage-per-test/utilities/tfc.py --lookup-and-print counts.index
Or choose another option.
Usage: tfc.py --index <dump file> <index file>
tfc.py --lookup|--lookup-and-print <index file>
Options:
-h, --help show this help message and exit
--index Make an index file from the coverage dump.
--lookup Accept a unified diff on stdin and return a regular
expression that will select the tests that should be
run.
--lookup-and-print Accept a unified diff on stdin and print out the
relevant tests that should be run.
--debug Show details of each step.
Notes
Running the full test suite with coverage options would be centralised and run continuously. Even so, it would always lag behind rocketfuel (unless PQM is very quiet), but I don't think that's critical. This tool is meant to be as accurate as possible, but doesn't need to be 100%.
The zope-coverage-per-test branch does not need to be merged into RF, it only needs to be merged into the centralised coverage service. The index file produced is quite small and could easily be downloaded regularly, say daily, to developer machines. Only the helper tool would be needed on developer machines for look-ups.
The look-up process could be provided as a web-service, and then no specific tools would be needed on developer machines. But this does not work if you're disconnected.
Comments
This looks very fun. -- CurtisHovey, 2007-11-22
Hey guys, this looks very cool. Did you think about taking a more static approach and just marking up files to indicate where their tests live? I'm not sure which approach I'd prefer, just thinking aloud. -- JonathanLange, 2007-11-27
Hi Jonathan. The association it records is from individual lines in source files back to the tests that exercise them, so too fine-grained to mark up the source code, though a coarser source file -> test index would be trivial to derive from that. I like the idea, especially because we would be reminded every time we open a source file of what tests hit it, so giving us an awareness of the codebase. Potentially good for learning and gardening. -- GavinPanella, 2007-11-27.
