= Change Launchpad to Use Chameleon ZPT Implementation in Lieu of Zope's =
Chameleon (http://chameleon.repoze.org/ and
https://bugs.launchpad.net/chameleon-template-engine) is a template
engine that re-implements Zope Page Templates. As
[[http://chameleon.repoze.org/docs/latest/|the documentation]] describes:
The engine compiles templates into Python byte-code. This results in
performance which is on average 10-15 times better than implementation
which use runtime interpretation. Real-world application benchmarks
show an overall performance improvement in complex applications of
30-50%.
These speed claims are repeated in other contexts; Plone developers, for
instance, report an increase in Plone (measured in requests per second rather than raw time
to serve) that ranges from
[[http://blog.hannosch.eu/2010/01/plone-4-how-much-faster-is-it.html|about 33% faster to more than 100% faster]].
'''Contact:''' Gary Poster < >
'''On Launchpad:''' https://launchpad.net/launchpad-project/+bugs?field.tag=chameleon
'''As a ''' Launchpad developer
'''I want ''' faster server-side generation across the board by switching to Chameleon
'''so that ''' Launchpad's users are happier and Launchpad's hardware resources are used more efficiently.
== Rationale ==
A Launchpad developer reports that tests seem to show that switching to
Chameleon can give us about a 15% speed increase on the server-side
generation time of representative pages. Other projects report a 33 to
100% speed increase on their representative pages by switching to
Chameleon.
A squad should need no more than a couple of weeks to iron out the
remaining test failures to see if Chameleon can make a difference for
us. The cost to benefit ratio at the start of the project might not
have been high enough to justify, but this proposal argues that it is
justifiable now.
Beyond the potential speed increase that Chameleon offers, Chameleon is also
more pluggable. For instance, the "cache:" tales expression we have now was
tales (tal:content="cache:...") and not tal (tal:cache="...") because it was
not easy to plug in new tal components. Chameleon supports this much better,
so we can contemplate making experiments in our templating language more
easily and more flexibly.
=== Details/History ===
Sidnei da Silva partially integrated Chameleon into Launchpad in 2008.
Tests never completely passed, and other projects were given priority
within the Launchpad team.
Gary Poster ran tests at the end of 2009 (on staging, turning off other
processes, comparing ten representative pages with and without
Chameleon; full results not available, unfortunately) that seemed to indicate
that Launchpad pages might get a typical win on the server of around 15%.
While that is a significant across-the-board improvement,
[[https://dev.launchpad.net/Foundations/SystemPerformance|other tests]]
indicated that bigger parts of our overall page speed were in the
network traffic, including SSL handshaking. Chameleon looked like
something to get to eventually, but not a high priority.
With the increased focus on server-side generation speed that Launchpad's TA
brought to the second half of 2010, Gary revisited that decision, and
decided it would be nice to try and pick up these changes. He investigated
the state of the incomplete Chameleon at that time and made some
progress, described in the "Status" section near the bottom of this document.
== Risks ==
Chameleon may not be measurably faster, or may have some killer problem, or may have enough smaller problems that they outweigh any incremental speed gain. We need to be aware that we might need to cut bait on this project. See the "Gotchas" listed in the Status section near the bottom of this proposal for known smaller problems as of this writing.
== Stakeholders ==
Robert Collins, Launchpad's TA, is regarded as the primary
decision maker as to whether we should pursue this proposal.
We should pursue this; jml will schedule it into the features queue once drafting of the LEP is complete. -- RobertCollins
== Constraints and Requirements ==
The change must support all current Launchpad features. It must be
measurably faster, and the encountered bugs must be either addressed or
documented and worked around.
== Success ==
Launchpad must have measurably faster server-side generation after , as
measured by our performance reports, for instance.
'''Bugs are at:''' [[https://launchpad.net/launchpad-project/+bugs?field.tag=chameleon|Bugs for this LEP]]
=== How will we know when we are done? ===
Chameleon is tested and rejected, or deployed and hardened.
=== How will we measure how well we have done? ===
We will use localized server-side performance reports on a selected production test
machine to see before-and-after performance. We should be able to
switch between Chameleon and Zope implementations with a simple
environment change and restart. This will give us the most numerically reliable
measurements.
We also ought to be able to see a change in the server-side performance
report (https://devpad.canonical.com/~lpqateam/ppr/) across the board
once Chameleon is deployed. This might give us more feel-good fuzzies.
== Thoughts? ==
=== Status ===
* Some aspects of Chameleon integration have been in the Launchpad tree since 2008.
* Gary has several branches that update Launchpad to the newest versions of Chameleon packages, implement Chameleon changes necessary for our memcache syntax, and fix a number of the test failures that occur when using Chameleon in Launchpad. The branches are as follows:
* lp:~gary/launchpad/chameleon
* lp:~gary/shipit/chameleon
* zope.app.apidoc : changes committed to trunk (svn+ssh://svn.zope.org/repos/main/zope.app.apidoc/trunk r118939) but not yet released ([[Bug:701623|needs more work for chameleon compatibility, either in zope.app.apidoc or in the Chameleon family]]).
In the course of working with Chameleon and Launchpad, Gary identified the following "gotchas" and changes to be aware of.
* Gotcha: Using "tal:define" to define a name that is also a Python builtin will not work well to pass values to macros--the builtin will win in the macro (e.g. "file" in productrelease-portlet-data.pt and "type" in zope.app.apidoc). Gary believes this can be corrected in z3c.pt, but has not attempted it. Bug:701627
* Gotcha: If you are writing an XML file with an XML header and do not incude an encoding, an encoding of "None" is added automatically. This can cause surprising errors. That said, you are always supposed to put an encoding in your xml file. lib/lp/testopenid/templates/application-xrds.pt lib/lp/testopenid/templates/persistentidentity-xrds.pt are affected by this. Bug:701632
* Gotcha: Using the tales "string:" prefix does not cast to a string if there are no other strings that are part of the string expression. See string:${announcement/id} in lp/registry/templates/announcement-macros.pt . Use Python for pure casts. This might be difficult to fix upstream. Bug:701633
* Gotcha: DOCTYPEs that are not at the start of the document confuse Chameleon. An example is {{{}}} that is in a metal section so it can be replaced--that's where I encountered it. The work-around: {{{}}} Bug:701643
* Gotcha: The tales expression "views/foo" is an old-fashioned spelling for "context/@@foo". It is not available in Chameleon's econtext. Replace the spelling. (This change was needed for apidoc only, and has been addressed .)
* Gotcha: Even while HTML 5 is arguably reducing the importance of XHTML, Chameleon increases XHTML strictness. Here are some examples encountered. (Bugs are not filed for these at this time.)
* Chameleon doesn't like stand-alone attrs (e.g.
in lib/lp/translations/templates/languageset-index.pt)
* Script must be escaped ( or
* All tags must be closed (e.g. and ).
* Gotcha: In some configurations, Chameleon generates .pt.py files. These must not be written in production, because the user running Launchpad does not have the privileges to write to the Launchpad code tree. The current Launchpad branch does not do this by default, and in fact Chameleon no longer does this by default, but it is something to be aware of. See also some of the comments in lp:~gary/launchpad/chameleon, in lp_sitecustomize.configure_chameleon. A bug is not filed for this at this time.
=== Rough suggested schedule ===
* 1 week to stomp on remaining bugs, get reviewed, and get merged.
* 1 week to get deployed on a single machine, get measurements to compare before and after performance, and to stomp on bugs.
* 1 week to get fully deployed and stomp on bugs.
* 1 week for things to Go Wrong.