= Help on Foundations Components = = Webservice = === Webservice integration in Launchpad === * This is done by annotating our Zope interfaces with lazr.restful annotations. * To understand the annotations, start with the annotations in `launchpad/lib/lp/bugs/interfaces/`, and also see the [[https://dev.launchpad.net/API/ImplementingAPIs|developer wiki documentation]]. * The implementation of the web service layer is in `lib/canonical/launchpad/webapp/servers.py`. See especially `WebServicePublication`, whose `getPrincipal` method implements OAuth authorization. * Most exceptions on startup are caused by annotation problems. Try finding the annotations you're using in `src/lazr/restful/declarations.py` and adding breakpoints. * If you need information, ask leonardr, benji, or gary. === Testing the webservice in Launchpad === * The `launchpadlib_for` method gives you a launchpadlib client object, which you can use in a unit test to browse your newly created factory resources. * If you need information, ask leonardr, benji, or gary. === lazr.restful === * Library to publish Zope interfaces as web service resources. * Used throughout Canonical, not just in Launchpad. * To understand it, look at the example service in `src/lazr/restful/example/base` * If you need information, ask leonardr, benji, or gary. === lazr.restfulclient === * Client-side library to consume WADL and JSON generated by lazr.restful. * Most of the functionality you think of as belonging to launchpadlib, actually belongs to lazr.restfulclient. * To understand it, look at the `Resource` class in `src/lazr/restfulclient/resource.py`. This class binds the JSON representation of a resource to its WADL representation and provides a Python interface to the data. * If you need information, ask leonardr, benji, or gary. === launchpadlib === * Client-side library to consume the Launchpad web service. * Includes only Launchpad-specific code. * To understand how to use it, see the [[https://help.launchpad.net/API|Launchpad help wiki]]. * Most of the code has to do with OAuth tokens. Look at `AuthorizeRequestTokenWithBrowser` to see how they're authorized, and look at `CredentialStore` and its subclasses to see how they're stored locally. Both classes are in `src/launchpadlib/credentials.py`. (Turning an OAuth token into a signed request is handled in lazr.restfulclient.) * If you need information, ask leonardr, benji, or gary. === wadllib === * Client-side library to parse WADL files into Python data structures. * Changes very slowly. * The best way to understand it is to read [[http://www.w3.org/Submission/wadl/|the WADL standard]], and to see how lazr.restfulclient uses wadllib. * If you need information, ask leonardr, benji, or gary. = Building and integrating dependencies = === Buildout === * See https://dev.launchpad.net/Hacking#Buildout . It gives all pointers to docs. * Gary, maybe Benji or Francis === meta-lp-debs and Launchpad's PPA === * See https://dev.launchpad.net/Hacking#Changing%20the%20launchpad%20dependency%20debs * In particular, see https://dev.launchpad.net/LaunchpadPpa . It describes meta-lp-debs and the procedure for updating it. * maxb, Gary, ... === Sourcecode === * Necessary for shipit; used by others * utilities/update-sourcecode in tree is the starting point for the code. * utilities/sourcedeps.conf in tree is the configuration * Gary, Curtis (?) = Testing and landing = === ec2 [test/land/...] script === * https://dev.launchpad.net/EC2Test * utilities/ec2 in tree * Jono, Gary, Ursula, Maris ... === Buildbot === * configuration is in its own project: https://launchpad.net/lpbuildbot * Maris, Gary === Tarmac and tarmac-land === * https://dev.launchpad.net/TarmacMergeMachinery * https://launchpad.net/bzr-tarmacland * Gary, Diogo, Ursula, Maris = Post-release QA and analysis = === OOPS tools === * Code is here: https://edge.launchpad.net/oops-tools/ * Installation instruction is in the README.txt file in the root of branch. * Deployment instructions: https://dev.launchpad.net/Foundations/QA/OopsToolsSetup * Diogo, Gary, Ursula, Maris === Performance reports (server side) === * Live here: https://devpad.canonical.com/~lpqateam/ppr/ * Generated by utilities/page-performance-report.py in the LP tree * Updates to the script are deployed on demand by losas * The report could do with javascript love to collapse unwanted sections and generate graphs on demand. This would let us have more graphs. * StuartBishop, Francis, Robert === Performance reports (client side) === * Live here: https://devpad.canonical.com/~mars/metrics/webpagetest.org/ * Measures what is arguably most painful for many of our users (OEM); Robert and James were going to investigate the possibility of proxies * See the [[https://launchpad.net/webpagetest.py|webpagetest.py project page]] for code and bugs. * Run from cron on devpad as ~mars * Maris === DB reports === * Live here: https://devpad.canonical.com/~lpqateam/dbr/ * Generated by utilities/report-database-stats.py in the LP tree * Run on wildcherry, so updated as part of a production database update * Reports made from snapshots generated by update_database_stats() in database/schema/trusted.sql * Can generate reports for arbitrary historical time periods to help diagnose load issues. * Most Read Tables isn't that meaningful (?) * StuartBishop === qa tagger and qa reports === * Code is here: https://launchpad.net/qa-tagger * Installation instruction in the README.txt file in the branch. * Hacking instructions in the HACKING.txt file in the branch. * More info about the process: https://dev.launchpad.net/QAProcessContinuousRollouts * Ursula, Diogo, Maris === ++profile++ === * See https://dev.launchpad.net/Debugging#Profiling%20page%20requests * Code is in lib/lp/services/profile/ * Gary, Maris, Benji = Authentication and identity = === openid === * Integration is in lib/lp/services/openid . Note we use one of the standard Python libraries for most of the guts. * The test/development OpenID provider is in LP and is different from the production provider. * The production OpenID provider is a Django app maintained by ISD: https://bugs.launchpad.net/canonical-identity-provider/ * A goal is to eventually be an OpenID RP (relying party, i.e., accept OpenID logins from other providers). See https://dev.launchpad.net/LEP/OpenIdRoadmap * See especially the "Thoughts" on the openid LEP mentioned above for a discussion of openid and LP identity * We use a testopenid server for devel (lib/lp/testopenid). It would be nice if we didn't have to use passwords any more on devel (Bug:629167, also in LEP). * We support more than one openid per person, though this is not exposed as well as we'd like. It solves some bugs, but needs more polish. See LEP for more. * No one is a full expert on this, but people involved have included Salgado, James Henstridge, Stuart, Gary, and Benji. === session db === * PostgreSQL backend for standard zope.app.session * Database connection runs in autocommit mode * StuartBishop wrote zope.app.session if you want help * Seperate PostgrerSQL database running in autocommit mode. * Librarian tokens use this storage area too because it was the most practical solution at the time. * Used by Launchpad to store authentication credentials and notifications (INotificationRequest) * Linked to browser cookie, so shared between browser windows. * Stealing the session cookie used to allow you to hijack the users authentication credentials. Not sure if this is still valid with OpenID. * Common PostgreSQL session db schema generated by database/schema/session.sql. Launchpad specific tweaks in database/schema/launchpad_session.sql. Kept separate in case we release as a separate component one day. * StuartBishop = other operational conveniences and tools = === haproxy pages (+opstats and the new one flacoste did) === = Feature flags = * https://dev.launchpad.net/LEP/FeatureFlags * Martin, Robert, Benji = Jobs system = (none of us did anything on this, but the page is in the Foundations section...) * https://dev.launchpad.net/Foundations/JobSystem * Aaron, Julian, ... = Javascript/Windmill = (?) * Maris, Paul, Deryck = Memcached = * In a LP ZPT extension, you can use tal:content="cache:..." to cache the rendered content. * lib/lp/services/memcache has the code. * lib/lp/services/memcache/doc/tales-cache.txt shows how they work. Note the testing behavior described in that document (in tests and dev mode, an HTML comment shows the settings of the cached value). * Note that the Chameleon work (see below) has a branch that changes this stuff significantly. Ideally you would see what the status is of this before proceeding. * Talk to Stuart, Gary * lib/lp/services/memcache/doc/restful-cache.txt describes functionality that is not used in LP (the ideas we had to invalidate data did not pan out). Talk to Leonard, Gary. = Chameleon = * Chameleon is an alternate page template implementation that is much faster than the standard ZPT implementation. It might bring around 15% speed increase across the board for server side changes. * Partly done Chameleon integration has been bit-rotting in LP for a long time. It is currently adding very little benefit, though the ++profile++ page uses it. (That could be relatively easily changed to use the standard implementation if we decided to rip Chameleon out). * See https://dev.launchpad.net/LEP/Chameleon for the current state of this project, including branches and gotchas. * Gary has done the most recent work on the LEP and associated branches; Sidnei did the original work.