Foundations/Webservice/Performance

Not logged in - Log In / Register

Revision 22 as of 2010-05-07 16:30:37

Clear message

This page tracks the work to discover and implement performance improvements to the Launchpad web service.

Solutions implemented

Request service root conditionally

Due to a bug in httplib2, launchpadlib was never making conditional requests for the service root even though the lazr.restfulclient tests worked. We changed the headers Launchpad serves and the problem went away.

Benefit: launchpadlib now downloads WADL only in very rare cases (when we upgrade Launchpad). Benefit accrues to existing launchpadlib installations.

In a live test, this reduced startup time from 3.9 seconds to 0.8 seconds.

Cache the service root client-side

We changed lazr.restful to serve Cache-Control headers along with the service root (WADL and JSON). For frozen versions of the web service (beta and 1.0) the Cache-Control max-age is one week; for devel it's one hour. We can tweak this further in the future.

Benefit: launchpadlib now makes HTTP requests on startup only once a week (or hour). Due to a bug in httplib2, benefit only accrues to installations with an up-to-date lazr.restfulclient.

Remove lazr.restfulclient's dependency on lazr.restful

This wasn't done for performance reasons, but it seems to be what brought launchpadlib import time from 0.36 seconds to 0.20 seconds due to time saved in pkg_resources.

Fix bug 568301

Named operations on collections were fetching the collections before invoking the operation. In a test against /edge, it took 13-19 seconds to invoke getByUniqueName on /branches. With the fix in place, it took about 1.3 seconds.

(This might not be completely fixed yet.)

Worthwhile but not implemented

Reinstate mod_compress on the 'api' vhost'

This will greatly reduce bandwidth usage, especially when serving WADL documents (1.2M to 100k). mod_compress was disabled in 2008 due to a bug which has since been fixed. We had a workaround in place, but it never worked.

Store representations in memcached

After constructing a JSON representation, we can store it in memory and use it later in preference to constructing it again.

Initial memcached experiment

I used the https://dev.launchpad.net/Foundations/Webservice?action=AttachFile&do=view&target=performance_test.py throughout my trials. For simplicity I'll only report the final number from that test, the mean end-to-end time across 30 fetches for a bug or for a collection of bugs.

First I hacked lazr.restful to serve representations directly from memcached, without any consideration for permission control. These results gives a theoretical lower bound on how fast we can can possibly make lazr.restful by using memcached.

Then I implemented a real representation caching system within lazr.restful, except using a Python dict instead of memcached. Here's how it works: if it turns out that a client can see the full representation of an entry (there are no redacted fields), the entry's string representation is stored in the cache. Upon subsequent requests, the string will be pulled from the cache and either served whole or incorporated into a JSON representation of a collection. If the client cannot see the full representation of a cached entry, we pull the cached string, simplejson.loads() it, redact the appropriate fields, and dump it again. A collection is built from (possibly cached) entry representations, using string operations rather than simplejson.dumps().

I then changed my implementation so that it used memcached instead of an in-memory Python dict, and ran the tests again. The results:

No representation cache: 0.09s (entry test), 0.74s (collection test) In-memory dict: 0.09s (entry test), 0.11s (collection test) memcached: 0.10s (entry test), 0.15s (collection test) Theoretical memcached best case: 0.09s (entry test), 0.18s (collection test)

In other words, with an in-memory dict it's about as fast to get one page of a collection as it is to get a single bug. Since memcached is slower than an in-memory dict, it's about 50% faster to get one bug than one page of a collection--but still much faster than not having a representation cache. The current memcached implementation is within the margin of error of the theoretical best case.

The main remaining problemis invalidation. My implementation invalidates an object's representation cache on an ObjectModifiedEvent (though this also needs a little work), but Launchpad isn't good about sending out ObjectModifiedEvents. We might get good results by doing cleanup on the Launchpad side and then tagging the corresponding objects as being okay to cache. Do this for bugs+bugtasks+people+branches and we'll have covered the most common access scenarios.

Not worthwhile/too much work

Speed up launchpadlib startup time

This is dominated by pkg_resources setup, so there's not that much we can do. We did improve this a bit by accident (see above).

Speed up wadllib parse time

I ran an experiment to see whether it would be faster to load the wadllib Application object from a pickle rather than parsing it every time. To get pickle to work I had to use elementtree.ElementTree (pure Python) instead of cElementtree. This made the initial parse go from about .3 seconds to about 3 seconds. Unpickling the pickle took about .63 seconds, twice the time it took to just parse the XML. It doesn't seem worth it. (Though I don't really see how it can be faster to create the Application from XML than from a pickle--maybe cElementtree is just really really fast.)

Cache collections in a noSQL database

Like MongoDB. The point of this story is to support keeping questions about collections from hitting postgres. That is much more expensive than just getting the values for a single row. If we can get the collections very fast from a noSQL db, that might be a big win. It would also support getting "nested" requests (see idea below) quickly. The proposed implementation is similar to the memcached story, except that triggers in postgres would completely maintain the pre-rendered data in the persistent noSQL db, rather than invalidating cached data. We would then use indexes in mongoDB to get the nested collections back. (The problem with this is we don't have good rules for collection cache invalidation.)

Use HTTP 1.1 KeepAlive

According to Gary, getting the Launchpad stack to support HTTP 1.1 is too risky: it fails in production under as-yet-unknown circumstances.

Ideas not tested yet

Cache entries and/or collections on the client side

When serving a representation of an entry or collection, send a Cache-Control header that specifies max-age. The client will not make subsequent requests for the entry until the max-age expires. Since the client requested this representation once, it's at least somewhat likely to do it again later. How much time this saves depends on what we choose for max-age.

Collections are currently never cached, but I think that's just because we don't serve any information that would let httplib2 make a conditional request or know when the cache would expire.

This is easy to implement, but to benefit we must accept some level of cache staleness on the client side. It has to be okay for a client to spend a while ignorant of some change to an entry, and (for a collection) ignorant of entries' addition to or removal from the collection.

Caching bug comments client-side is an clear win, since they never change, but it's rare for a client to request a specific bug comment, so it's a very small win. Caching the _collection_ of a bug's comments would be a much bigger win, but then clients would go a while without knowing that a new comment was added to a bug, which we don't like.

Because we are so insistent on providing up-to-date data, I believe the scope for this solution is very small. It's possible some of the HWDB resources could use this.

Collection-specific ETags

We have no general way of calculating an ETag or a Last-Modified for a collection, but it might be possible to set up a hook method that calculates these values for specific collections. This would let launchpadlib make conditional requests for collections. _This_ would let us cache a bug's comments on the client side.

Others

Process

First step: quantify performance

We want to be able to measure our performance. Ideally, this would be both end-to-end and subdivided into our network performance, our performance on the client, and our performance on the server. These have four goals.

The last goal means we need to find a balance between thoroughness and expediency in our construction of tests.

Second step: collect, evaluate, winnow, and prioritize possible solutions

We are particularly responsible for the systemic performance of the webservice. This means that we want the average performance to be good. We need to work with the Launchpad team to create good performance within individual requests, but we are more interested here with things that can make the whole webservice faster. Tools that can help developers make individual pages faster easily, but with some effort and customization, are also of interest.

Again, our solutions will focus on different aspects of the end-to-end performance of the webservice. We then have three basic areas to attack.

Third step: implement the next solution

The next solution is TBD.

Next...

Rinse and repeat back to the first step, trying to determine if our quantifiable performance gives an qualitative experience that we find acceptable.