Rendering of reStructured text is not possible, please install Docutils.
=========================
Long-polling in Launchpad
=========================
.. contents::
Overview
========
In order that browsers may receive as-it-happens updates of server
processes that run asynchronously with respect to the web application
- of which there are many in Launchpad - we're using a combination of
a message queue, RabbitMQ_, a AMQP to HTTP bridge, txlongpoll_, some
additional abstraction/convenience APIs within Launchpad, and a
JavaScript component that terminates the event pipeline in the
browser.
This document talks mostly about the APIs within Launchpad, but will
also touch on RabbitMQ_, txlongpoll_, and the `JavaScript
component`_.
.. _RabbitMQ: http://www.rabbitmq.com/
.. _txlongpoll: https://launchpad.net/txlongpoll
.. _`JavaScript component`: /JavaScript
Here is an overview of all the components involved in long-polling in
Launchpad::
+------------+ LP.cache.longpoll.{uri, key}
| LP |<-----------(1)-----------------+
+------------+ |
^ v
+-----------+ | +--------------------------+
| |<---LongPollEvent-(3)-+ | Browser |
| RabbitMQ | |--------------------------|
| Server | | |
| |<---RabbitEvent---(4)-+ | longpolljs (JSEvent (6))|
+-----------+ | +--------------------------+
v ^
+------------+ /{uri}?uuid={key}&sequence=0 |
| txlongpoll |<------------(2)----------------+
+------------+ --(5)->
<-(7)--
The workflow for a typical usage of the long-poll system will go
something like this:
1) The LP web server serves a web page with ``LP.cache.longpoll.uri``
and ``LP.cache.longpoll.key`` populated. If these are not
populated, the long-poll machinery will stop here.
2) The JavaScript long-poll library then uses these to initiate a long
lasting XHR request to the Twisted-based ``txlongpoll`` server.
3) Some time later, something noteworthy happens in Launchpad, and the
code that wants to tell the world obtains an object providing
``ILongPollEvent``, typically via adaption. It then uses
``long_poll_event.emit(**payload)`` to send a message out via
RabbitMQ, keyed with ``long_poll_event.event_key``.
4) The event will be routed by RabbitMQ out to listeners in
``txlongpoll``.
5) The long lasting XHR request to the ``txlongpoll`` server returns
(with the JSONified ``payload`` that was passed to
``long_poll_event.emit(...)``).
6) The JavaScript long-poll library then propagates this payload via
YUI's event system so that any piece of in-page code can receive
events from server-side.
7) The JavaScript long-poll library now reconnects to ``txlongpoll``
to await additional events.
Long-poll's usage
=================
- `Merge proposal's diff update`_: merge proposal's diffs are updated
via XHR as soon as new version is available (no more page reloads!)
.. _`Merge proposal's diff update`: /MPDiffUpdate
RabbitMQ
========
RabbitMQ_ is the messaging system used to connect events between
discrete components. In this case it connects long running jobs, that
emit an event when finishing, to the txlongpoll_ server which informs
the browser.
txlongpoll
==========
`lp:txlongpoll`_ accepts HTTP requests from browsers and holds them
open (for ~4-5 minutes at a time) while waiting for messages to arrive
from the messaging system, to which it connects via AMQP.
.. _lp:txlongpoll: https://launchpad.net/txlongpoll
It is based on Twisted_ and is a completely separate service to
Launchpad; it runs stand-alone and does not depend on Launchpad in any
way.
.. _Twisted: http://twistedmatrix.com/
Long-poll request format
========================
txlongpoll_ awaits HTTP GET requests of the form
``/uuid={queue_name}&sequence={sequence_id}`` The ``uuid`` specifies a
queue that should already exist and be subscribed (bound) to events
from an exchange within RabbitMQ_. The ``sequence`` is a sequential
integer that txlongpoll_ requires and is used to better support
load-balancing. It is incremented before each request.
----
Writing code on the server side
===============================
LongPollEvent
-------------
`LongPollEvent`_ is the abstract adapter base class that should be
used to create custom event adapters.
**Why?** -- *Julian*
All that's needed is an implementation of ``ILongPollEvent``;
subclassing ``LongPollEvent`` is not *required*. However, it has a
sensible implementation of ``emit`` that is either usable as-is,
or worth reusing via calling-up from subclasses. -- *Gavin*
.. _LongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/event.py
Sub-classes need to define the `event_key` property and declare
something along the lines of::
class LongPollAwesomeThingEvent(LongPollEvent):
adapts(IAwesomeThing)
implements(ILongPollEvent)
..
**What is event_key for?** -- *Julian*
See `ILongPollEvent`_ ;) It is the key with which events will be
emitted. It should be predictable and stable. This is the key to
which queues can be subscribed. An example could be
``longpoll.event.branchmergeproposal.1234``. -- *Gavin*
.. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py
Alternatively, use the `long_poll_event` class decorator::
@long_poll_event(IAwesomeThing)
class LongPollAwesomeThingEvent(LongPollEvent):
...
In both cases the adapter should be registered in a `configure.zcml`
somewhere sensible::
<adapter factory=".adapters.LongPollAwesomeThingEvent" />
See `LongPollStormEvent`_ and its supporting code for a cargo-cultable
example of this, and read on for an explanation of it.
.. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py
zope.event bridge for Storm life-cycle events
---------------------------------------------
As a convenience, subscription handlers for `life-cycle events`_ of
Storm model objects already exist. They generate long-poll events
named ``longpoll.event.{table_name}.{object_primary_key}`` with the
type of event ("created", "deleted" or "modified") in the event
payload.
.. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py
For instance, if ``storm_object`` is an object from a table named
``FakeTable`` with a primary id of 1234, the following call::
notify(ObjectDeletedEvent(storm_object))
... will result in an event being fired with the following data::
{"event_key": "longpoll.event.faketable.1234",
"what": "deleted"})
ILongPollSubscriber
-------------------
``ILongPollSubscriber`` describes a subscriber to long-poll events. It
defines a ``subscribe_key`` property:
"The key which the subscriber must know in order to be able to
long-poll for subscribed events. Should be infeasible to guess, a
UUID for example."
An example could be
``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``.
It also defines a ``subscribe`` method that can be passed an object
that provides ``ILongPollEvent``. In the back-end this ends up
creating a queue named ``subscribe_key`` which is bound to event's
``event_key`` (not exclusively; it can be bound to multiple events).
As the time of this writing only a request (as in
``IApplicationRequest``) can be used as a subscriber using
LongPollApplicationRequestSubscriber_
**I don't understand what a "request" and a "subscriber" is in this
context** -- *Julian*
I think I've clarified that above. -- *Gavin*
.. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py
This adapter ensures that the request's JSON cache is stuffed with the
necessary information that the JavaScript long-poll component needs to
operate.
A request can be subscribed to events quite simply::
ILongPollSubscriber(request).subscribe(FakeEvent())
However, there's an even easier way, using
``lp.app.longpoll.subscribe``::
subscribe(FakeEvent())
Or, when ``some_object`` can be adapted to ``ILongPollEvent`` this
reduces even further::
subscribe(some_object)
ILongPollSubscriber for Storm
-----------------------------
If ``model_object`` is a Storm model object (i.e. is an instance of
``Storm``) then a simple ``subscribe(model_object)`` call arranges all
of the following:
1. Declares a new subscribe queue with an unguessable name (see
``subscribe_key``).
2. Binds the subscribe queue to the ``event_key`` defined by the
``ILongPollEvent`` adapter of ``some_object``.
3. Puts the ``subscribe_key`` in the request's JSON cache as
``longpoll.key``, sets ``longpoll.uri`` so that the in-page
JavaScript knows where to connect, and adds the ``subscribe_key``
to the ``longpoll.subscriptions`` list.
Testing
=======
To test that your code triggers the appropriate long-poll events, see
`LongPollEventRecord`_ and the context manager
`capture_longpoll_emissions`_::
from lp.services.longpoll.testing import (
capture_longpoll_emissions,
LongPollEventRecord,
)
with capture_longpoll_emissions() as log:
pass
# Here something should trigger a long poll event.
expected = LongPollEventRecord(
"my-event-key", {
"event_key": "my-event-key",
"something": "somedata"})
self.assertEqual([expected], log)
.. _`LongPollEventRecord`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py
.. _`capture_longpoll_emissions`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py
Job done
========
That's the back-end plumbing, now it's up to the in-page JavaScript to
know what to do with the events that it'll be able to long-poll for...
Next: `/JavaScript`_ 
launchpad development