launchpad development
Help on Foundations Components
Webservice
Webservice integration in Launchpad
- This is done by annotating our Zope interfaces with lazr.restful annotations.
To understand the annotations, start with the annotations in launchpad/lib/lp/bugs/interfaces/, and also see the developer wiki documentation.
The implementation of the web service layer is in lib/canonical/launchpad/webapp/servers.py. See especially WebServicePublication, whose getPrincipal method implements OAuth authorization.
Most exceptions on startup are caused by annotation problems. Try finding the annotations you're using in src/lazr/restful/declarations.py and adding breakpoints.
- If you need information, ask leonardr, benji, or gary.
Testing the webservice in Launchpad
The launchpadlib_for method gives you a launchpadlib client object, which you can use in a unit test to browse your newly created factory resources.
- If you need information, ask leonardr, benji, or gary.
lazr.restful
- Library to publish Zope interfaces as web service resources.
- Used throughout Canonical, not just in Launchpad.
To understand it, look at the example service in src/lazr/restful/example/base
- If you need information, ask leonardr, benji, or gary.
lazr.restfulclient
- Client-side library to consume WADL and JSON generated by lazr.restful.
- Most of the functionality you think of as belonging to launchpadlib, actually belongs to lazr.restfulclient.
To understand it, look at the Resource class in src/lazr/restfulclient/resource.py. This class binds the JSON representation of a resource to its WADL representation and provides a Python interface to the data.
- If you need information, ask leonardr, benji, or gary.
launchpadlib
- Client-side library to consume the Launchpad web service.
- Includes only Launchpad-specific code.
To understand how to use it, see the Launchpad help wiki.
Most of the code has to do with OAuth tokens. Look at AuthorizeRequestTokenWithBrowser to see how they're authorized, and look at CredentialStore and its subclasses to see how they're stored locally. Both classes are in src/launchpadlib/credentials.py. (Turning an OAuth token into a signed request is handled in lazr.restfulclient.)
- If you need information, ask leonardr, benji, or gary.
wadllib
- Client-side library to parse WADL files into Python data structures.
- Changes very slowly.
The best way to understand it is to read the WADL standard, and to see how lazr.restfulclient uses wadllib.
- If you need information, ask leonardr, benji, or gary.
Building and integrating dependencies
Buildout
See https://dev.launchpad.net/Hacking#Buildout . It gives all pointers to docs.
- Gary, maybe Benji or Francis
meta-lp-debs and Launchpad's PPA
See https://dev.launchpad.net/Hacking#Changing%20the%20launchpad%20dependency%20debs
In particular, see https://dev.launchpad.net/LaunchpadPpa . It describes meta-lp-debs and the procedure for updating it.
- maxb, Gary, ...
Sourcecode
- Necessary for shipit; used by others
- utilities/update-sourcecode in tree is the starting point for the code.
- utilities/sourcedeps.conf in tree is the configuration
- Gary, Curtis (?)
Testing and landing
ec2 [test/land/...] script
- utilities/ec2 in tree
- Jono, Gary, Ursula, Maris ...
Buildbot
configuration is in its own project: https://launchpad.net/lpbuildbot
- Maris, Gary
Tarmac and tarmac-land
- Gary, Diogo, Ursula, Maris
Post-release QA and analysis
OOPS tools
Code is here: https://edge.launchpad.net/oops-tools/
- Installation instruction is in the README.txt file in the root of branch.
Deployment instructions: https://dev.launchpad.net/Foundations/QA/OopsToolsSetup
- Diogo, Gary, Ursula, Maris
Performance reports (server side)
Live here: https://devpad.canonical.com/~lpqateam/ppr/
- Generated by utilities/page-performance-report.py in the LP tree
- Updates to the script are deployed on demand by losas
- The report could do with javascript love to collapse unwanted sections and generate graphs on demand. This would let us have more graphs.
StuartBishop, Francis, Robert
Performance reports (client side)
Live here: https://devpad.canonical.com/~mars/metrics/webpagetest.org/
- Measures what is arguably most painful for many of our users (OEM); Robert and James were going to investigate the possibility of proxies
See the webpagetest.py project page for code and bugs.
- Run from cron on devpad as ~mars
- Maris
DB reports
Live here: https://devpad.canonical.com/~lpqateam/dbr/
- Generated by utilities/report-database-stats.py in the LP tree
- Run on wildcherry, so updated as part of a production database update
- Reports made from snapshots generated by update_database_stats() in database/schema/trusted.sql
- Can generate reports for arbitrary historical time periods to help diagnose load issues.
- Most Read Tables isn't that meaningful (?)
qa tagger and qa reports
Code is here: https://launchpad.net/qa-tagger
- Installation instruction in the README.txt file in the branch.
- Hacking instructions in the HACKING.txt file in the branch.
More info about the process: https://dev.launchpad.net/QAProcessContinuousRollouts
- Ursula, Diogo, Maris
++profile++
See https://dev.launchpad.net/Debugging#Profiling%20page%20requests
- Code is in lib/lp/services/profile/
- Gary, Maris, Benji
Authentication and identity
openid
- Integration is in lib/lp/services/openid . Note we use one of the standard Python libraries for most of the guts.
- The test/development OpenID provider is in LP and is different from the production provider.
The production OpenID provider is a Django app maintained by ISD: https://bugs.launchpad.net/canonical-identity-provider/
A goal is to eventually be an OpenID RP (relying party, i.e., accept OpenID logins from other providers). See https://dev.launchpad.net/LEP/OpenIdRoadmap
- See especially the "Thoughts" on the openid LEP mentioned above for a discussion of openid and LP identity
We use a testopenid server for devel (lib/lp/testopenid). It would be nice if we didn't have to use passwords any more on devel (629167, also in LEP).
- We support more than one openid per person, though this is not exposed as well as we'd like. It solves some bugs, but needs more polish. See LEP for more.
- No one is a full expert on this, but people involved have included Salgado, James Henstridge, Stuart, Gary, and Benji.
session db
- PostgreSQL backend for standard zope.app.session
- Database connection runs in autocommit mode
StuartBishop wrote zope.app.session if you want help
- Seperate PostgrerSQL database running in autocommit mode.
- Librarian tokens use this storage area too because it was the most practical solution at the time.
- Used by Launchpad to store authentication credentials and notifications (INotificationRequest)
- Linked to browser cookie, so shared between browser windows.
- Stealing the session cookie used to allow you to hijack the users authentication credentials. Not sure if this is still valid with OpenID.
- Common PostgreSQL session db schema generated by database/schema/session.sql. Launchpad specific tweaks in database/schema/launchpad_session.sql. Kept separate in case we release as a separate component one day.
other operational conveniences and tools
haproxy pages (+opstats and the new one flacoste did)
Feature flags
- Martin, Robert, Benji
Jobs system
(none of us did anything on this, but the page is in the Foundations section...)
- Aaron, Julian, ...
Javascript/Windmill
(?)
- Maris, Paul, Deryck
Memcached
- In a LP ZPT extension, you can use tal:content="cache:..." to cache the rendered content.
- lib/lp/services/memcache has the code.
- lib/lp/services/memcache/doc/tales-cache.txt shows how they work. Note the testing behavior described in that document (in tests and dev mode, an HTML comment shows the settings of the cached value).
- Note that the Chameleon work (see below) has a branch that changes this stuff significantly. Ideally you would see what the status is of this before proceeding.
- Talk to Stuart, Gary
- lib/lp/services/memcache/doc/restful-cache.txt describes functionality that is not used in LP (the ideas we had to invalidate data did not pan out). Talk to Leonard, Gary.
Chameleon
- Chameleon is an alternate page template implementation that is much faster than the standard ZPT implementation. It might bring around 15% speed increase across the board for server side changes.
- Partly done Chameleon integration has been bit-rotting in LP for a long time. It is currently adding very little benefit, though the ++profile++ page uses it. (That could be relatively easily changed to use the standard implementation if we decided to rip Chameleon out).
See https://dev.launchpad.net/LEP/Chameleon for the current state of this project, including branches and gotchas.
- Gary has done the most recent work on the LEP and associated branches; Sidnei did the original work.
