## page was renamed from TestsFromCode ||This page is outdated, and is here for reference only.|| = Change Launchpad, Find Tests = [[JonathanKnowles|jsk]] had a dream of a day when a diff could be used to find the exact set of tests to run, thus giving him mucho confidence before submitting a branch for review, but without the long wait to run `make check`, and without needing a [[BjornTillenius|Tillenius]]-like understanding of the codebase and her mysterious ways. And so it came to pass... [[JonathanKnowles|jsk]] and [[GavinPanella|I]] did some early work on such a tool at AllHands (2007). I did some more work on it after that, and now have a prototype: {{{ bzr+ssh://devpad.canonical.com/code/gavin.panella/launchpad/zope-coverage-per-test }}} == Uses == 1. Navigating the code base. * For someone new to the code this will help them discover the tests relevant to their area. * Our tests are also documentation, so this index also helps us find relevant documentation. 2. Increasing ''velocity''. Because we can quickly identify the tests we need to run, we can: * avoid a lengthy test run before submitting code for review, * reduce the likelihood of unforeseen issues in PQM when we don't have time for a full test run beforehand, * and thus reduce the likelihood of subsequent re-review. == Getting started == There are two parts to this tool: 1. [[#Recording|Recording]]: * A patch to the Zope tree to let us record coverage information for individual tests. * A tool to create an index from that coverage information. 2. [[#Lookup|Lookup]]: * A tool that uses the index to find tests relevant to changes in a development branch of Launchpad. If you just want to see how this might be useful to you as a developer, skip to [[#Lookup|step 2]]. <> == Recording == The branch has an example set-up script, `utilities/tfc-setup.sh` (also [[attachment:tfc-setup.sh|here]]). This will create you a new LP branch with a customised `sourcecode/zope`. Once set-up, there are two steps: 1. Run the test suite with coverage options. This means running `bin/test` with `--coverage=` and (new) `--coverage-per-test`. This creates a dump file containing tests and the source files (and lines) they exercise. 2. Run `utilities/tfc.py --index` to create an index file which will let us map from source files+lines back to the tests. === Example === Run the test suite, recording coverage. {{{ $ bin/test -vvt lib/canonical/ --coverage=coverage --coverage-per-test Running tests at level 1 Running unit tests: ... Total: 10850 tests, 0 failures, 0 errors }}} Make the index. {{{ $ ./sourcecode/zope/utilities/tfc.py --index coverage/counts.{dump,index} $ du -h coverage/* 98M coverage/counts.dump 396K coverage/counts.index }}} <> == Lookup == Pull the branch. The tool you need is `utilities/tfc.py`. You also need the index file from the [[#Recording|recording]] phase. (!) Conveniently, I've baked an index already: [[attachment:counts.index]]. It'll be out of date by the time you try this, but will probably be good enough as an example. Next you just run `utilities/tfc.py --lookup` to figure out the list of the tests we need to run. This actually returns a regular expression which can be used as the `-t` argument to `bin/test`. === Example === Get a copy of the `zope-coverage-per-test` branch. {{{ bzr branch bzr+ssh://devpad.canonical.com/code/gavin.panella/launchpad/zope-coverage-per-test }}} Work in a new `rocketfuel` branch. {{{ bzr branch rocketfuel tfc-example-branch cd tfc-example-branch }}} Make changes to the source tree. For example, add an extra space to line 81 of lib/canonical/widgets/project.py: {{{ sed -i '81s/==/== /' lib/canonical/widgets/project.py }}} Download the index file. {{{ wget -O counts.index --user=warthogs --password=${password:?} \ 'https://launchpad.canonical.com/TestsFromChanges?action=AttachFile&do=get&target=counts.index' }}} Now lookup the relevant tests. {{{ bzr diff | ../zope-coverage-per-test/utilities/tfc.py --lookup-and-print counts.index }}} Or choose another option. {{{ Usage: tfc.py --index tfc.py --lookup|--lookup-and-print Options: -h, --help show this help message and exit --index Make an index file from the coverage dump. --lookup Accept a unified diff on stdin and return a regular expression that will select the tests that should be run. --lookup-and-print Accept a unified diff on stdin and print out the relevant tests that should be run. --debug Show details of each step. }}} == Notes == Running the full test suite with coverage options would be centralised and run continuously. Even so, it would always lag behind rocketfuel (unless PQM is very quiet), but I don't think that's critical. This tool is meant to be as accurate as possible, but doesn't need to be 100%. The zope-coverage-per-test branch does not need to be merged into RF, it only needs to be merged into the centralised coverage service. The index file produced is quite small and could easily be downloaded regularly, say daily, to developer machines. Only the helper tool would be needed on developer machines for look-ups. The look-up process could be provided as a web-service, and then no specific tools would be needed on developer machines. But this does not work if you're disconnected. == Comments == This looks very fun. -- CurtisHovey, 2007-11-22 Hey guys, this looks very cool. Did you think about taking a more static approach and just marking up files to indicate where their tests live? I'm not sure which approach I'd prefer, just thinking aloud. -- JonathanLange, 2007-11-27 Hi Jonathan. The association it records is from individual lines in source files back to the tests that exercise them, so too fine-grained to mark up the source code, though a coarser source file -> test index would be trivial to derive from that. I like the idea, especially because we would be reminded every time we open a source file of what tests hit it, so giving us an awareness of the codebase. Potentially good for learning and gardening. -- GavinPanella, 2007-11-27.