1748
Comment:
|
5772
setup and teardown aren't that expensive
|
Deletions are marked like this. | Additions are marked like this. |
Line 6: | Line 6: |
== How is Windmill integrated? == | <<TableOfContents()>> |
Line 8: | Line 8: |
There is a script called `lp-windmill.py` in the top-level directory. This is a wrapper around the windmill main script which starts a Launchpad server on port 8085 locally with a fresh database (including all of the standard tests sample data). |
== Using Windmill to Test Launchpad == |
Line 13: | Line 10: |
The windmill process is then fired off with the command line argument. | === Setting up Windmill === |
Line 15: | Line 12: |
So if you want to run all the tests you'd typically use: {{{ $ ./lp-windmill.py -t lib/canonical/launchpad/windmill/tests firefox http://launchpad.dev:8085 }}} |
Windmill is included in the Launchpad source tree. Its dependencies are installed by the `launchpad-developer-dependencies` ubuntu package. |
Line 22: | Line 15: |
For interactive test running and development, it's usually more convenient to run the Launchpad server separately: |
=== Running the Launchpad Windmill test suite === To run all Windmill tests: |
Line 26: | Line 20: |
$ ./lp-windmill.py --server-only }}} |
bin/test --layer=WindmillLayer }}} |
Line 29: | Line 23: |
And then fire off a windmill session using the normal command: | To run all Windmill tests for a single application (e.g. the Bugs application): |
Line 32: | Line 26: |
$ ./utilities/windmill.py shell firefox http://launchpad.dev:8085 }}} |
bin/test --layer=BugsWindmillLayer }}} |
Line 35: | Line 29: |
That way you can restart windmill, without having to wait for the Launchpad restart cycle. |
To run a single Windmill test (e.g. test_bug_commenting() in the Bugs application): |
Line 38: | Line 32: |
See the [[http://trac.getwindmill.com/wiki/BookChapter-2-RunningWindmill#TheShellEnvironment|help page]] on the shell environment. |
{{{ bin/test --layer=BugsWindmillLayer --test=test_bug_commenting }}} The Windmill tests are integrated into our normal test suite, and the tests has a layered named `FooWindmillLayer`, where `Foo` is the name of the application (e.g. `BugsWindmillLayer`. The Windmill layers are excluded by default, so when running a Windmill tests you always have to specify the `--layer` option, otherwise the test runner won't find it. |
Line 45: | Line 43: |
rooted at `lib/canonical/launchpad/windmill/tests`. Tests are divided by applications and then subdivided by workflows. |
placed in `lib/lp/<application>/windmill/tests`. |
Line 53: | Line 49: |
== External resources == | |
Line 55: | Line 50: |
* [[http://trac.getwindmill.com/wiki/WindmillBook|Windmill Book]] | = Test Writing Tips = * Test the Happy Path, and one or two error paths * Edge cases are best pushed into the unit tests * Prefer element ids to XPath for locating page elements * XPath makes your tests brittle and dependent on page structure * Try to pull test actions out into their own functions, creating a Python Windmill testing [[http://en.wikipedia.org/wiki/Domain-specific_programming_language|DSL]] * Use the [[http://code.google.com/p/webdriver/wiki/PageObjects|PageObject pattern]] to encapsulate the actions available through specific pages * '''Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIXMLHttpRequest.setRequestHeader]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" ''' -- you likely started your windmill shell using the wrong hostname, ie. `launchpad.dev` instead of `bugs.launchpad.dev` When using `open()`, make sure to use `waits.forPageLoad()` before any `waits.forElement()` (unless you are absolutely sure that you can't be on a page that already contains that element). When using the lazr-js slide-in/slide-out effects, use the following assertions to test whether the element is visible or not. {{{#! python # Test for visibility. client.asserts.assertProperty( id=u'search-results', validator='className|lazr-opened') # Test for invisibility. client.asserts.assertProperty( id=u'search-results', validator='className|lazr-closed') }}} When hiding or revealing elements using POJS, use the `unseen` class to control this behavior. It's easy to test for too. E.g. {{{#! javascript // Hide this element. element1.addClass('unseen'); // Reveal this element. element2.removeClass('unseen'); }}} {{{#! python # Test for visibility. client.asserts.assertNotProperty( id=u'search-results', validator='className|unseen') # Test for invisibility. client.asserts.assertProperty( id=u'search-results', validator='className|unseen') }}} = External resources = * [[http://trac.getwindmill.com/wiki#Documentation|Windmill Book]] * [[http://www.slideshare.net/alexchaffee/fullstack-webapp-testing-with-selenium-and-rails|Full-stack webapp testing with Selenium and Rails]] - has a number of tips that apply to any automated testing framework = Troubleshooting = * If you are running on a slow computer or inside a VM, you may randomly trigger timeout errors. One way around these errors is to run each test individually: {{{ $ bin/test --layer=WindmillLayer --list-tests | sed -e '1d' > tests.list $ cat tests.list | xargs -i bin/test --layer=WindmillLayer --test='{}' >> test.log }}} * In the Windmill IDE window, you can test windmill's ability to find objects by going to the Tools menu (not the Firefox Tools menu) and clicking on "Firebug Lite". In Firebug Lite, you will be able to call the lookupNode() function with the same locator parameters that windmill uses. For example: * {{{ lookupNode( {classname: 'yui-picker'} ) }}} * {{{ lookupNode( {xpath: '//table[@id=foo]//button'} ) }}} * [[http://trac.getwindmill.com/wiki/ControllerApi|Locator Docs]] * Run `make schema` if you're having an issue that ends in something like: {{{ File "/usr/lib/python2.4/urllib.py", line 82, in urlopen return opener.open(url) File "/usr/lib/python2.4/urllib.py", line 190, in open return getattr(self, name)(url) File "/usr/lib/python2.4/urllib.py", line 316, in open_http errcode, errmsg, headers = h.getreply() File "/usr/lib/python2.4/httplib.py", line 1137, in getreply response = self._conn.getresponse() File "/usr/lib/python2.4/httplib.py", line 866, in getresponse response.begin() File "/usr/lib/python2.4/httplib.py", line 336, in begin version, status, reason = self._read_status() File "/usr/lib/python2.4/httplib.py", line 294, in _read_status line = self.fp.readline() File "/usr/lib/python2.4/socket.py", line 317, in readline data = recv(1) IOError: [Errno socket error] (104, 'Connection reset by peer') }}} ---- CategoryJavaScript CategoryTesting |
Windmill for JS Integration Tests
For integration testing that covers JS workflows, our tool of choice is Windmill.
Contents
Using Windmill to Test Launchpad
Setting up Windmill
Windmill is included in the Launchpad source tree. Its dependencies are installed by the launchpad-developer-dependencies ubuntu package.
Running the Launchpad Windmill test suite
To run all Windmill tests:
bin/test --layer=WindmillLayer
To run all Windmill tests for a single application (e.g. the Bugs application):
bin/test --layer=BugsWindmillLayer
To run a single Windmill test (e.g. test_bug_commenting() in the Bugs application):
bin/test --layer=BugsWindmillLayer --test=test_bug_commenting
The Windmill tests are integrated into our normal test suite, and the tests has a layered named FooWindmillLayer, where Foo is the name of the application (e.g. BugsWindmillLayer. The Windmill layers are excluded by default, so when running a Windmill tests you always have to specify the --layer option, otherwise the test runner won't find it.
How are the tests organized
Tests written in python are placed in lib/lp/<application>/windmill/tests.
Tests using the JS API are in lib/canonical/launchpad/windmill/jstests.
Test Writing Tips
- Test the Happy Path, and one or two error paths
- Edge cases are best pushed into the unit tests
- Prefer element ids to XPath for locating page elements
- XPath makes your tests brittle and dependent on page structure
Try to pull test actions out into their own functions, creating a Python Windmill testing DSL
Use the PageObject pattern to encapsulate the actions available through specific pages
Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIXMLHttpRequest.setRequestHeader]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" -- you likely started your windmill shell using the wrong hostname, ie. launchpad.dev instead of bugs.launchpad.dev
When using open(), make sure to use waits.forPageLoad() before any waits.forElement() (unless you are absolutely sure that you can't be on a page that already contains that element).
When using the lazr-js slide-in/slide-out effects, use the following assertions to test whether the element is visible or not.
# Test for visibility. client.asserts.assertProperty( id=u'search-results', validator='className|lazr-opened') # Test for invisibility. client.asserts.assertProperty( id=u'search-results', validator='className|lazr-closed')
When hiding or revealing elements using POJS, use the unseen class to control this behavior. It's easy to test for too. E.g.
// Hide this element. element1.addClass('unseen'); // Reveal this element. element2.removeClass('unseen');
# Test for visibility. client.asserts.assertNotProperty( id=u'search-results', validator='className|unseen') # Test for invisibility. client.asserts.assertProperty( id=u'search-results', validator='className|unseen')
External resources
Full-stack webapp testing with Selenium and Rails - has a number of tips that apply to any automated testing framework
Troubleshooting
- If you are running on a slow computer or inside a VM, you may randomly trigger timeout errors. One way around these errors is to run each test individually:
$ bin/test --layer=WindmillLayer --list-tests | sed -e '1d' > tests.list $ cat tests.list | xargs -i bin/test --layer=WindmillLayer --test='{}' >> test.log
- In the Windmill IDE window, you can test windmill's ability to find objects by going to the Tools menu (not the Firefox Tools menu) and clicking on "Firebug Lite". In Firebug Lite, you will be able to call the lookupNode() function with the same locator parameters that windmill uses. For example:
lookupNode( {classname: 'yui-picker'} )
lookupNode( {xpath: '//table[@id=foo]//button'} )
Run make schema if you're having an issue that ends in something like:
File "/usr/lib/python2.4/urllib.py", line 82, in urlopen return opener.open(url) File "/usr/lib/python2.4/urllib.py", line 190, in open return getattr(self, name)(url) File "/usr/lib/python2.4/urllib.py", line 316, in open_http errcode, errmsg, headers = h.getreply() File "/usr/lib/python2.4/httplib.py", line 1137, in getreply response = self._conn.getresponse() File "/usr/lib/python2.4/httplib.py", line 866, in getresponse response.begin() File "/usr/lib/python2.4/httplib.py", line 336, in begin version, status, reason = self._read_status() File "/usr/lib/python2.4/httplib.py", line 294, in _read_status line = self.fp.readline() File "/usr/lib/python2.4/socket.py", line 317, in readline data = recv(1) IOError: [Errno socket error] (104, 'Connection reset by peer')