Windmill for JS Integration Tests

For integration testing that covers JS workflows, our tool of choice is Windmill.

Using Windmill to Test Launchpad

Setting up Windmill

Windmill is included in the Launchpad source tree. Its dependencies are installed by the launchpad-developer-dependencies ubuntu package.

Running the Launchpad Windmill test suite

There is a script called lp-windmill.py in the top-level directory. This is a wrapper around the windmill main script which starts a Launchpad server on port 8085 locally with a fresh database (including all of the standard tests sample data).

The windmill process is then fired off with the command line argument.

So if you want to run all the tests you'd typically use:

• $ ./bin/lp-windmill -e test=lib/canonical/launchpad/windmill/tests/<test_module> firefox http://launchpad.dev:8085

  Note that it is not currently possible to run all the tests like that because of a bug/limitation in Windmill. You need to run each tests starting at the correct vhost. So for bugs tests, you'd use:

  $ ./bin/lp-windmill -e test=lib/canonical/launchpad/windmill/tests/test_bugs firefox http://bugs.launchpad.dev:8085

  and for registry, you'd use:

  $ ./bin/lp-windmill -e test=lib/canonical/launchpad/windmill/tests/test_registry firefox http://launchpad.dev:8085

For interactive test running and development, it's usually more convenient to run the Launchpad server separately:

• $ ./bin/lp-windmill --server-only

The Windmill driver script, windmill.py, is located under the utilities/ directory. Using it, you can start an interactive shell to run tests without setting up and tearing down Launchpad repeatedly:

• $ ./bin/windmill shell firefox http://launchpad.dev:8085

See the help page on the shell environment.

How are the tests organized

Tests written in python are rooted at lib/canonical/launchpad/windmill/tests.

Tests are divided by applications and then subdivided by workflows.

Tests using the JS API are in lib/canonical/launchpad/windmill/jstests.

Test Writing Tips

• Test the Happy Path, and one or two error paths
• Edge cases are best pushed into the unit tests
• Setup and teardown are expensive, so there will be a trend towards testing more per test
• Prefer element ids to XPath for locating page elements
  • XPath makes your tests brittle and dependent on page structure

• Try to pull test actions out into their own functions, creating a Python Windmill testing DSL

• Use the PageObject pattern to encapsulate the actions available through specific pages

• Error: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIXMLHttpRequest.setRequestHeader]" nsresult: "0x80004005 (NS_ERROR_FAILURE)" -- you likely started your windmill shell using the wrong hostname, ie. launchpad.dev instead of bugs.launchpad.dev

When using the lazr-js slide-in/slide-out effects, use the following assertions to test whether the element is visible or not.

    # Test for visibility.
    client.asserts.assertProperty(
        id=u'search-results',
        validator='className|lazr-opened')

    # Test for invisibility.
    client.asserts.assertProperty(
        id=u'search-results',
        validator='className|lazr-closed')

When hiding or revealing elements using POJS, use the unseen class to control this behavior. It's easy to test for too. E.g.

    // Hide this element.
    element1.addClass('unseen');
    // Reveal this element.
    element2.removeClass('unseen');

    # Test for visibility.
    client.asserts.assertNotProperty(
        id=u'search-results',
        validator='className|unseen')

    # Test for invisibility.
    client.asserts.assertProperty(
        id=u'search-results',
        validator='className|unseen')

External resources

• Windmill Book

• Full-stack webapp testing with Selenium and Rails - has a number of tips that apply to any automated testing framework

Troubleshooting

• If you are running on a slow computer or inside a VM, you may randomly trigger timeout errors.
• In the Windmill IDE window, you can test windmill's ability to find objects by going to the Tools menu (not the Firefox Tools menu) and clicking on "Firebug Lite". In Firebug Lite, you will be able to call the lookupNode() function with the same locator parameters that windmill uses. For example:

  •  lookupNode( {classname: 'yui-picker'} ) 

  •  lookupNode( {xpath: '//table[@id=foo]//button'} ) 

  • Locator Docs

CategoryJavaScript CategoryTesting