Diff for "LongPoll"

Differences between revisions 16 and 26 (spanning 10 versions)
Rendering of reStructured text is not possible, please install Docutils.

=========================
Long-polling in Launchpad
=========================

.. contents::


Overview
========

In order that browsers may receive as-it-happens updates of server
processes that run asynchronously with respect to the web application
- of which there are many in Launchpad - we're using a combination of
a message queue, RabbitMQ_, a AMQP to HTTP bridge, txlongpoll_, some
additional abstraction/convenience APIs within Launchpad, and a
JavaScript component that terminates the event pipeline in the
browser.

This document talks mostly about the APIs within Launchpad, but will
also touch on RabbitMQ_, txlongpoll_, and the `JavaScript
component`_.

.. _RabbitMQ: http://www.rabbitmq.com/
.. _txlongpoll: https://launchpad.net/txlongpoll
.. _`JavaScript component`: /JavaScript

Here is an overview of all the components involved in long-polling in
Launchpad::

                              +------------+  LP.cache.longpoll.{uri, key}
                              |     LP     |<-----------(1)-----------------+
                              +------------+                                |
                                    ^                                       v
 +-----------+                      |                           +--------------------------+
 |           |<---LongPollEvent-(3)-+                           |        Browser           |
 | RabbitMQ  |                                                  |--------------------------|
 |  Server   |                                                  |                          |
 |           |<---RabbitEvent---(4)-+                           | longpolljs  (JSEvent (6))|
 +-----------+                      |                           +--------------------------+                   
                                    v                                       ^
                              +------------+  /{uri}?uuid={key}&sequence=0  |
                              | txlongpoll |<------------(2)----------------+
                              +------------+           --(5)->
                                                       <-(7)--

The workflow for a typical usage of the long-poll system will go
something like this:

1) The LP web server serves a web page with ``LP.cache.longpoll.uri``
   and ``LP.cache.longpoll.key`` populated. If these are not
   populated, the long-poll machinery will stop here.
2) The JavaScript long-poll library then uses these to initiate a long
   lasting XHR request to the Twisted-based ``txlongpoll`` server.
3) Some time later, something noteworthy happens in Launchpad, and the
   code that wants to tell the world obtains an object providing
   ``ILongPollEvent``, typically via adaption. It then uses
   ``long_poll_event.emit(**payload)`` to send a message out via
   RabbitMQ, keyed with ``long_poll_event.event_key``.
4) The event will be routed by RabbitMQ out to listeners in
   ``txlongpoll``.
5) The long lasting XHR request to the ``txlongpoll`` server returns
   (with the JSONified ``payload`` that was passed to
   ``long_poll_event.emit(...)``).
6) The JavaScript long-poll library then propagates this payload via
   YUI's event system so that any piece of in-page code can receive
   events from server-side.
7) The JavaScript long-poll library now reconnects to ``txlongpoll``
   to await additional events.


Long-poll's usage
=================

- `Merge proposal's diff update`_: merge proposal's diffs are updated
  via XHR as soon as new version is available (no more page reloads!)

.. _`Merge proposal's diff update`: /MPDiffUpdate 


RabbitMQ
========

RabbitMQ_ is the messaging system used to connect events between
discrete components. In this case it connects long running jobs, that
emit an event when finishing, to the txlongpoll_ server which informs
the browser.


txlongpoll
==========

`lp:txlongpoll`_ accepts HTTP requests from browsers and holds them
open (for ~4-5 minutes at a time) while waiting for messages to arrive
from the messaging system, to which it connects via AMQP.

.. _lp:txlongpoll: https://launchpad.net/txlongpoll

It is based on Twisted_ and is a completely separate service to
Launchpad; it runs stand-alone and does not depend on Launchpad in any
way.

.. _Twisted: http://twistedmatrix.com/


Long-poll request format
========================

txlongpoll_ awaits HTTP GET requests of the form
``/uuid={queue_name}&sequence={sequence_id}`` The ``uuid`` specifies a
queue that should already exist and be subscribed (bound) to events
from an exchange within RabbitMQ_. The ``sequence`` is a sequential
integer that txlongpoll_ requires and is used to better support
load-balancing. It is incremented before each request.

----


Writing code on the server side
===============================


LongPollEvent
-------------

`LongPollEvent`_ is the abstract adapter base class that should be
used to create custom event adapters.

  **Why?** -- *Julian*

    All that's needed is an implementation of ``ILongPollEvent``;
    subclassing ``LongPollEvent`` is not *required*. However, it has a
    sensible implementation of ``emit`` that is either usable as-is,
    or worth reusing via calling-up from subclasses. -- *Gavin*

.. _LongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/event.py

Sub-classes need to define the `event_key` property and declare
something along the lines of::

 class LongPollAwesomeThingEvent(LongPollEvent):
     adapts(IAwesomeThing)
     implements(ILongPollEvent)

..

  **What is event_key for?** -- *Julian*

    See `ILongPollEvent`_ ;) It is the key with which events will be
    emitted. It should be predictable and stable. This is the key to
    which queues can be subscribed. An example could be
    ``longpoll.event.branchmergeproposal.1234``. -- *Gavin*

.. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py

Alternatively, use the `long_poll_event` class decorator::

 @long_poll_event(IAwesomeThing)
 class LongPollAwesomeThingEvent(LongPollEvent):
     ...

In both cases the adapter should be registered in a `configure.zcml`
somewhere sensible::

 <adapter factory=".adapters.LongPollAwesomeThingEvent" />

See `LongPollStormEvent`_ and its supporting code for a cargo-cultable
example of this, and read on for an explanation of it.

.. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py


zope.event bridge for Storm life-cycle events
---------------------------------------------

As a convenience, subscription handlers for `life-cycle events`_ of
Storm model objects already exist. They generate long-poll events
named ``longpoll.event.{table_name}.{object_primary_key}`` with the
type of event ("created", "deleted" or "modified") in the event
payload.

.. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py

For instance, if ``storm_object`` is an object from a table named
``FakeTable`` with a primary id of 1234, the following call::

 notify(ObjectDeletedEvent(storm_object))

... will result in an event being fired with the following data::

 {"event_key": "longpoll.event.faketable.1234",
  "what": "deleted"})


ILongPollSubscriber
-------------------

``ILongPollSubscriber`` describes a subscriber to long-poll events. It
defines a ``subscribe_key`` property:

  "The key which the subscriber must know in order to be able to
  long-poll for subscribed events. Should be infeasible to guess, a
  UUID for example."

An example could be
``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``.

It also defines a ``subscribe`` method that can be passed an object
that provides ``ILongPollEvent``. In the back-end this ends up
creating a queue named ``subscribe_key`` which is bound to event's
``event_key`` (not exclusively; it can be bound to multiple events).

As the time of this writing only a request (as in
``IApplicationRequest``) can be used as a subscriber using
LongPollApplicationRequestSubscriber_

  **I don't understand what a "request" and a "subscriber" is in this
  context** -- *Julian*

    I think I've clarified that above. -- *Gavin*

.. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py

This adapter ensures that the request's JSON cache is stuffed with the
necessary information that the JavaScript long-poll component needs to
operate.

A request can be subscribed to events quite simply::

 ILongPollSubscriber(request).subscribe(FakeEvent())

However, there's an even easier way, using
``lp.app.longpoll.subscribe``::

  subscribe(FakeEvent())

Or, when ``some_object`` can be adapted to ``ILongPollEvent`` this
reduces even further::

  subscribe(some_object)


ILongPollSubscriber for Storm
-----------------------------

If ``model_object`` is a Storm model object (i.e. is an instance of
``Storm``) then a simple ``subscribe(model_object)`` call arranges all
of the following:

1. Declares a new subscribe queue with an unguessable name (see
   ``subscribe_key``).

2. Binds the subscribe queue to the ``event_key`` defined by the
   ``ILongPollEvent`` adapter of ``some_object``.

3. Puts the ``subscribe_key`` in the request's JSON cache as
   ``longpoll.key``, sets ``longpoll.uri`` so that the in-page
   JavaScript knows where to connect, and adds the ``subscribe_key``
   to the ``longpoll.subscriptions`` list.


Testing
=======

To test that your code triggers the appropriate long-poll events, see
`LongPollEventRecord`_ and the context manager
`capture_longpoll_emissions`_::

 from lp.services.longpoll.testing import (
     capture_longpoll_emissions,
     LongPollEventRecord,
     )

 with capture_longpoll_emissions() as log:
     pass
     # Here something should trigger a long poll event.

 expected = LongPollEventRecord(
     "my-event-key", {
         "event_key": "my-event-key",
         "something": "somedata"})

 self.assertEqual([expected], log)

.. _`LongPollEventRecord`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py

.. _`capture_longpoll_emissions`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py


Job done
========

That's the back-end plumbing, now it's up to the in-page JavaScript to
know what to do with the events that it'll be able to long-poll for...

Next: `/JavaScript`_
launchpad development
