Where to get it

bzr branch lp:lp-dev-utils

How to install

This is a standard buildout project; it has a makefile for convenience:

make bin/buildout
bin/buildout

And then you can run any script using bin/py:

bin/py page-performance-report.py

tools

There are a bunch of tools contained in lp-dev-utils, including line of code counters, log analysers, commercial project management API scripts and so forth.

page-performance-report

The page performance report is one of the key tools used by Launchpad to gather an aggregate view on application performance. It can run in two modes: zope trace log, and apache log mode - these just determine the parser used to obtain the timing data for requests. The key functional difference is that apache log files don't include backend SQL or other processing data, so the information in the report is less rich.

The report contains various processing of urls, to group or otherwise categorise them. For each grouping, key statistics like the 99th percentile are calculated (which is a good indicator of user experience, unlike the average). A small graph is presented as well, which gives a visual representation of the timing data for that category. We find this very useful for zeroing in on slow or misbehaving pages, even when users don't tell us. (Though they often do).

Note: the %D time in Apache log files indicates total time until Apache stops handling the request, so a slow client or network will give longer times. Placing haproxy or a similar front end accelerator that will suck up small pages in their entirety will mitigate this (as well as giving you solid load balancing :)).

To use the page performance report, there are five steps:

Install lp-dev-utils as shown above
Link jquery into the output directory you will be using. (We'll rewrite this to use YUI in the future, probably the next time someone has new features to add).
Change your apache log to include the time it takes to serve the response:
```
LogFormat "%h %l %u %t \"%r\" %>s %b %D"
```
Create a customised ini file that maps your URL's to categories / templates / etc.

Cron it all up. Add a cron entry like

cd path-to-lp-dev-utils && PPR_SWITCHES=--no-partition CONFIG_FILE=my-ppr.ini LOGS_ROOTS="/path/to/log/dirs" LOG_PATTERN="login\*access\*" OUTPUT_ROOT=/srv/dev.mysite.com/html/ JQUERY_URL=/urlfragment/to/js/dir ./page-performance-report-daily.sh

The PPR outputs a number of html files. We run it daily, weekly and monthly - see the example shell scripts (which are what Launchpad itself runs with).