Diff for "LongPoll"

Not logged in - Log In / Register

Differences between revisions 3 and 26 (spanning 23 versions)
Revision 3 as of 2011-09-26 10:38:55
Size: 9022
Editor: rvb
Comment:
Revision 26 as of 2011-10-04 20:06:01
Size: 10940
Editor: allenap
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
========
LongPoll
========
=========================
Long-polling in Launchpad
=========================

.. contents::
Line 10: Line 13:
Here is an overview of all the components involved in long polling::
In order that browsers may receive as-it-happens updates of server
processes that run asynchronously with respect to the web application
- of which there are many in Launchpad - we're using a combination of
a message queue, RabbitMQ_, a AMQP to HTTP bridge, txlongpoll_, some
additional abstraction/convenience APIs within Launchpad, and a
JavaScript component that terminates the event pipeline in the
browser.

This document talks mostly about the APIs within Launchpad, but will
also touch on RabbitMQ_, txlongpoll_, and the `JavaScript
component`_.

.. _RabbitMQ: http://www.rabbitmq.com/
.. _txlongpoll: https://launchpad.net/txlongpoll
.. _`JavaScript component`: /JavaScript

Here is an overview of all the components involved in long-polling in
Launchpad::
Line 29: Line 48:
The workflow for a typical usage of the LongPoll system of LP will go like this:

1) The LP web server serves a web page with LP.cache.longpoll.uri and LP.cache.longpoll.key populated. (If these are not populated, the LongPoll machinery will stop here)
2) The javascript longpoll library then uses these to initiate a long lasting XHR request to the twisted based LongPoll server.
3) (Some time later) The LP app server uses LongPollEvent.emit(payload) to send a message to the RabbitMQ server (if the server is available)
4) The event will then be published by RabbitMQ to the txlongpoll server.
5) The long lasting XHR request to the txlongpoll server returns (with the JSONified payload from the LongPollEvent).
6) This will be handled by the javascript longpoll library which will trigger a javascript event with the payload.
7) The javascript longpoll library will now reconnect to be able to handle other events.

LongPoll server
================

See `lp:txlongpoll`_ for a project that describes a so-called long-polling server. It is Twisted-based and maps HTTP requests to consumption of messages on Rabbit queues.
The workflow for a typical usage of the long-poll system will go
something like this:

1) The LP web server serves a web page with ``LP.cache.longpoll.uri``
   and ``LP.cache.longpoll.key`` populated. If these are not
   populated, the long-poll machinery will stop here.
2) The JavaScript long-poll library then uses these to initiate a long
   lasting XHR request to the Twisted-based ``txlongpoll`` server.
3) Some time later, something noteworthy happens in Launchpad, and the
   code that wants to tell the world obtains an object providing
   ``ILongPollEvent``, typically via adaption. It then uses
   ``long_poll_event.emit(**payload)`` to send a message out via
   RabbitMQ, keyed with ``long_poll_event.event_key``.
4) The event will be routed by RabbitMQ out to listeners in
   ``txlongpoll``.
5) The long lasting XHR request to the ``txlongpoll`` server returns
   (with the JSONified ``payload`` that was passed to
   ``long_poll_event.emit(...)``).
6) The JavaScript long-poll library then propagates this payload via
   YUI's event system so that any piece of in-page code can receive
   events from server-side.
7) The JavaScript long-poll library now reconnects to ``txlongpoll``
   to await additional events.


Long-poll's usage
=================

- `Merge proposal's diff update`_: merge proposal's diffs are updated
  via XHR as soon as new version is available (no more page reloads!)

.. _`Merge proposal's diff update`: /MPDiffUpdate


RabbitMQ
========

RabbitMQ_ is the messaging system used to connect events between
discrete components. In this case it connects long running jobs, that
emit an event when finishing, to the txlongpoll_ server which informs
the browser.


txlongpoll
==========

`lp:txlongpoll`_ accepts HTTP requests from browsers and holds them
open (for ~4-5 minutes at a time) while waiting for messages to arrive
from the messaging system, to which it connects via AMQP.
Line 46: Line 100:
This means we can have Javascript in the browser doing XHR to the LongPoll server and block on long-running jobs until the job emits a message saying it's done. There is currently work in progress to make merge proposal diffs appear as soon as the job completes so that the end user doesn't have to refresh the MP page.

LongPoll request format
It is based on Twisted_ and is a completely separate service to
Launchpad; it runs stand-alone and does not depend on Launchpad in any
way.

.. _Twisted: http://twistedmatrix.com/


Long-poll request format
Line 51: Line 110:
The LongPoll server awaits connections of the form '/uuid={queue_name}&sequence={sequence_id}'. The uuid parameter will be used to connect to a specific queue. The sequence parameter is a sequential integer that RabbitMQ requires to fetch successive events from the same queue.

LongPoll events
===============
txlongpoll_ awaits HTTP GET requests of the form
``/uuid={queue_name}&sequence={sequence_id}`` The ``uuid`` specifies a
queue that should already exist and be subscribed (bound) to events
from an exchange within RabbitMQ_. The ``sequence`` is a sequential
integer that txlongpoll_ requires and is used to better support
load-balancing. It is incremented before each request.

----


Writing code on the server side
===============================
Line 59: Line 127:
`LongPollEvent`_ is the abstract adapter base class that should be used to create a custom event adapters. `LongPollEvent`_ is the abstract adapter base class that should be
used to create custom event adapters.

  **Why?** -- *Julian*

    All that's needed is an implementation of ``ILongPollEvent``;
    subclassing ``LongPollEvent`` is not *required*. However, it has a
    sensible implementation of ``emit`` that is either usable as-is,
    or worth reusing via calling-up from subclasses. -- *Gavin*
Line 63: Line 139:
Sub-classes need to define the `event_key` property and declare something along the lines of:: Sub-classes need to define the `event_key` property and declare
something along the lines of::
Line 69: Line 146:
..

  **What is event_key for?** -- *Julian*

    See `ILongPollEvent`_ ;) It is the key with which events will be
    emitted. It should be predictable and stable. This is the key to
    which queues can be subscribed. An example could be
    ``longpoll.event.branchmergeproposal.1234``. -- *Gavin*

.. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py
Line 75: Line 163:
In both cases the adapter should be registered in a `configure.zcml` somewhere sensible:: In both cases the adapter should be registered in a `configure.zcml`
somewhere sensible::
Line 79: Line 168:
zope.event bridge
-----------------

As a convenience, adapters for the storm events ObjectCreatedEvent, ObjectDeletedEvent and ObjectModifiedEvent already exist. They generate events named "longpoll.event.{table_name}.{object_primary_key}" with the type of event ("created", "deleted" or "modified") in the payload.

For instance calling, if storm_object is an object from a table named FakeTable with a primary id of 1234::
See `LongPollStormEvent`_ and its supporting code for a cargo-cultable
example of this, and read on for an explanation of it.

.. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py


zope.event bridge for Storm life-cycle events
---------------------------------------------

As a convenience, subscription handlers for `life-cycle events`_ of
Storm model objects already exist. They generate long-poll events
named ``longpoll.event.{table_name}.{object_primary_key}`` with the
type of event ("created", "deleted" or "modified") in the event
payload.

.. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py

For instance, if ``storm_object`` is an object from a table named
``FakeTable`` with a primary id of 1234, the following call::
Line 88: Line 190:
Will result in this event being fired:: ... will result in an event being fired with the following data::
Line 93: Line 195:

ILongPollSubscriber
-------------------

``ILongPollSubscriber`` describes a subscriber to long-poll events. It
defines a ``subscribe_key`` property:

  "The key which the subscriber must know in order to be able to
  long-poll for subscribed events. Should be infeasible to guess, a
  UUID for example."

An example could be
``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``.

It also defines a ``subscribe`` method that can be passed an object
that provides ``ILongPollEvent``. In the back-end this ends up
creating a queue named ``subscribe_key`` which is bound to event's
``event_key`` (not exclusively; it can be bound to multiple events).

As the time of this writing only a request (as in
``IApplicationRequest``) can be used as a subscriber using
LongPollApplicationRequestSubscriber_

  **I don't understand what a "request" and a "subscriber" is in this
  context** -- *Julian*

    I think I've clarified that above. -- *Gavin*

.. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py

This adapter ensures that the request's JSON cache is stuffed with the
necessary information that the JavaScript long-poll component needs to
operate.

A request can be subscribed to events quite simply::

 ILongPollSubscriber(request).subscribe(FakeEvent())

However, there's an even easier way, using
``lp.app.longpoll.subscribe``::

  subscribe(FakeEvent())

Or, when ``some_object`` can be adapted to ``ILongPollEvent`` this
reduces even further::

  subscribe(some_object)


ILongPollSubscriber for Storm
-----------------------------

If ``model_object`` is a Storm model object (i.e. is an instance of
``Storm``) then a simple ``subscribe(model_object)`` call arranges all
of the following:

1. Declares a new subscribe queue with an unguessable name (see
   ``subscribe_key``).

2. Binds the subscribe queue to the ``event_key`` defined by the
   ``ILongPollEvent`` adapter of ``some_object``.

3. Puts the ``subscribe_key`` in the request's JSON cache as
   ``longpoll.key``, sets ``longpoll.uri`` so that the in-page
   JavaScript knows where to connect, and adds the ``subscribe_key``
   to the ``longpoll.subscriptions`` list.

Line 94: Line 264:
-------

To test that your code triggers the appropriate long poll events, use the class `LongPollEventRecord`_ and the context manager `capture_longpoll_emissions`_::
=======

To test that your code triggers the appropriate long-poll events, see
`LongPollEventRecord`_ and the context manager
`capture_longpoll_emissions`_::
Line 112: Line 284:
 self.assertEqual(expected, log)  self.assertEqual([expected], log)
Line 118: Line 290:
LongPoll subscription
======================

As the time of this writing only a request can be used as a subscriber using LongPollApplicationRequestSubscriber_

.. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py

Once a LongPollApplicationRequestSubscriber object is created, individual events can be subscribed to it::

 subscriber = LongPollApplicationRequestSubscriber(request)
 subscriber.subscribe(FakeEvent())

After the first event has been subscribed, LP.cache.longpoll.key will be populated with a key (identifying the RabbitMQ queue) specific to this request.

LongPoll Javascript library
============================

The `long poll Javascript library`_ is responsible for connecting to the long poll server and firing Javascript events to reflect the state of this connection and make RabbitMQ event available to the Javascript layer.

.. _`long poll Javascript library`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/app/longpoll/javascript/longpoll.js

Connection
----------

The long poll library will issue a connection to the long poll server iff LP.cache.longpoll.uri and LP.cache.longpoll.key are populated in the JSON cache.
After each successful or failed connection, the long poll library reconnects to the long poll server to be able to handle other events.

Events
------

Two kind of events are fired by the long poll Javascript library. Events deriving from LongPollEvents and "system" events indicating the state of the long polling connection.

- event_key: fired each time RabbitMQ passed an event created using the LongPollEvent adapter.

For instance, this in LP python code::

 # subscription code omitted.
 class FakeEvent(LongPollEvent):

     implements(ILongPollEvent)

     @property
     def event_key(self):
        return "event-key-test"

 event = FakeEvent("source")
 event_data = {"hello": 1234}
 event.emit(**event_data)

Will result in an event being fired on the Javascript side::

 Y.fire("event-key-test", {"hello": 1234});

- 'lp.app.longpoll.failure': fired each time a long poll transaction fails. This might be because of connection problems or because the library failed to parse the event payload.

To avoid hammering the server in case of an outage, the Long poll Javascript library waits for 1 second before reconnecting. Also, after 5 failed connections, the library will wait 3 minutes before trying to reconnect. These events can be used to make the end user aware of the connection problem.

- 'lp.app.longpoll.longdelay': fired each time the library switches in 'longdelay' mode (i.e. after 5 failed connections).

- 'lp.app.longpoll.shortdelay': fired each time the library switches back in 'shortdelay' mode (i.e. after at least 5 failed connections and then after a successful connection).

Testing
-------

Since all the long poll Javascript library will be doing is firing Javascript events, testing a Javascript application that interacts with this library is pretty straightforward. Simply fire the events and make sure the application reacts appropriately::

 // Fire "event-key-test".
 Y.fire("event-key-test", {"hello": 1234});

 // Simulate a failure.
 Y.fire("lp.app.longpoll.failure");

 // Simulate the library going into 'longdelay' mode.
 Y.fire("lp.app.longpoll.longdelay");

Job done
========

That's the back-end plumbing, now it's up to the in-page JavaScript to
know what to do with the events that it'll be able to long-poll for...

Next: `/JavaScript`_
Rendering of reStructured text is not possible, please install Docutils.

=========================
Long-polling in Launchpad
=========================

.. contents::


Overview
========

In order that browsers may receive as-it-happens updates of server
processes that run asynchronously with respect to the web application
- of which there are many in Launchpad - we're using a combination of
a message queue, RabbitMQ_, a AMQP to HTTP bridge, txlongpoll_, some
additional abstraction/convenience APIs within Launchpad, and a
JavaScript component that terminates the event pipeline in the
browser.

This document talks mostly about the APIs within Launchpad, but will
also touch on RabbitMQ_, txlongpoll_, and the `JavaScript
component`_.

.. _RabbitMQ: http://www.rabbitmq.com/
.. _txlongpoll: https://launchpad.net/txlongpoll
.. _`JavaScript component`: /JavaScript

Here is an overview of all the components involved in long-polling in
Launchpad::

                              +------------+  LP.cache.longpoll.{uri, key}
                              |     LP     |<-----------(1)-----------------+
                              +------------+                                |
                                    ^                                       v
 +-----------+                      |                           +--------------------------+
 |           |<---LongPollEvent-(3)-+                           |        Browser           |
 | RabbitMQ  |                                                  |--------------------------|
 |  Server   |                                                  |                          |
 |           |<---RabbitEvent---(4)-+                           | longpolljs  (JSEvent (6))|
 +-----------+                      |                           +--------------------------+                   
                                    v                                       ^
                              +------------+  /{uri}?uuid={key}&sequence=0  |
                              | txlongpoll |<------------(2)----------------+
                              +------------+           --(5)->
                                                       <-(7)--

The workflow for a typical usage of the long-poll system will go
something like this:

1) The LP web server serves a web page with ``LP.cache.longpoll.uri``
   and ``LP.cache.longpoll.key`` populated. If these are not
   populated, the long-poll machinery will stop here.
2) The JavaScript long-poll library then uses these to initiate a long
   lasting XHR request to the Twisted-based ``txlongpoll`` server.
3) Some time later, something noteworthy happens in Launchpad, and the
   code that wants to tell the world obtains an object providing
   ``ILongPollEvent``, typically via adaption. It then uses
   ``long_poll_event.emit(**payload)`` to send a message out via
   RabbitMQ, keyed with ``long_poll_event.event_key``.
4) The event will be routed by RabbitMQ out to listeners in
   ``txlongpoll``.
5) The long lasting XHR request to the ``txlongpoll`` server returns
   (with the JSONified ``payload`` that was passed to
   ``long_poll_event.emit(...)``).
6) The JavaScript long-poll library then propagates this payload via
   YUI's event system so that any piece of in-page code can receive
   events from server-side.
7) The JavaScript long-poll library now reconnects to ``txlongpoll``
   to await additional events.


Long-poll's usage
=================

- `Merge proposal's diff update`_: merge proposal's diffs are updated
  via XHR as soon as new version is available (no more page reloads!)

.. _`Merge proposal's diff update`: /MPDiffUpdate 


RabbitMQ
========

RabbitMQ_ is the messaging system used to connect events between
discrete components. In this case it connects long running jobs, that
emit an event when finishing, to the txlongpoll_ server which informs
the browser.


txlongpoll
==========

`lp:txlongpoll`_ accepts HTTP requests from browsers and holds them
open (for ~4-5 minutes at a time) while waiting for messages to arrive
from the messaging system, to which it connects via AMQP.

.. _lp:txlongpoll: https://launchpad.net/txlongpoll

It is based on Twisted_ and is a completely separate service to
Launchpad; it runs stand-alone and does not depend on Launchpad in any
way.

.. _Twisted: http://twistedmatrix.com/


Long-poll request format
========================

txlongpoll_ awaits HTTP GET requests of the form
``/uuid={queue_name}&sequence={sequence_id}`` The ``uuid`` specifies a
queue that should already exist and be subscribed (bound) to events
from an exchange within RabbitMQ_. The ``sequence`` is a sequential
integer that txlongpoll_ requires and is used to better support
load-balancing. It is incremented before each request.

----


Writing code on the server side
===============================


LongPollEvent
-------------

`LongPollEvent`_ is the abstract adapter base class that should be
used to create custom event adapters.

  **Why?** -- *Julian*

    All that's needed is an implementation of ``ILongPollEvent``;
    subclassing ``LongPollEvent`` is not *required*. However, it has a
    sensible implementation of ``emit`` that is either usable as-is,
    or worth reusing via calling-up from subclasses. -- *Gavin*

.. _LongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/event.py

Sub-classes need to define the `event_key` property and declare
something along the lines of::

 class LongPollAwesomeThingEvent(LongPollEvent):
     adapts(IAwesomeThing)
     implements(ILongPollEvent)

..

  **What is event_key for?** -- *Julian*

    See `ILongPollEvent`_ ;) It is the key with which events will be
    emitted. It should be predictable and stable. This is the key to
    which queues can be subscribed. An example could be
    ``longpoll.event.branchmergeproposal.1234``. -- *Gavin*

.. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py

Alternatively, use the `long_poll_event` class decorator::

 @long_poll_event(IAwesomeThing)
 class LongPollAwesomeThingEvent(LongPollEvent):
     ...

In both cases the adapter should be registered in a `configure.zcml`
somewhere sensible::

 <adapter factory=".adapters.LongPollAwesomeThingEvent" />

See `LongPollStormEvent`_ and its supporting code for a cargo-cultable
example of this, and read on for an explanation of it.

.. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py


zope.event bridge for Storm life-cycle events
---------------------------------------------

As a convenience, subscription handlers for `life-cycle events`_ of
Storm model objects already exist. They generate long-poll events
named ``longpoll.event.{table_name}.{object_primary_key}`` with the
type of event ("created", "deleted" or "modified") in the event
payload.

.. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py

For instance, if ``storm_object`` is an object from a table named
``FakeTable`` with a primary id of 1234, the following call::

 notify(ObjectDeletedEvent(storm_object))

... will result in an event being fired with the following data::

 {"event_key": "longpoll.event.faketable.1234",
  "what": "deleted"})


ILongPollSubscriber
-------------------

``ILongPollSubscriber`` describes a subscriber to long-poll events. It
defines a ``subscribe_key`` property:

  "The key which the subscriber must know in order to be able to
  long-poll for subscribed events. Should be infeasible to guess, a
  UUID for example."

An example could be
``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``.

It also defines a ``subscribe`` method that can be passed an object
that provides ``ILongPollEvent``. In the back-end this ends up
creating a queue named ``subscribe_key`` which is bound to event's
``event_key`` (not exclusively; it can be bound to multiple events).

As the time of this writing only a request (as in
``IApplicationRequest``) can be used as a subscriber using
LongPollApplicationRequestSubscriber_

  **I don't understand what a "request" and a "subscriber" is in this
  context** -- *Julian*

    I think I've clarified that above. -- *Gavin*

.. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py

This adapter ensures that the request's JSON cache is stuffed with the
necessary information that the JavaScript long-poll component needs to
operate.

A request can be subscribed to events quite simply::

 ILongPollSubscriber(request).subscribe(FakeEvent())

However, there's an even easier way, using
``lp.app.longpoll.subscribe``::

  subscribe(FakeEvent())

Or, when ``some_object`` can be adapted to ``ILongPollEvent`` this
reduces even further::

  subscribe(some_object)


ILongPollSubscriber for Storm
-----------------------------

If ``model_object`` is a Storm model object (i.e. is an instance of
``Storm``) then a simple ``subscribe(model_object)`` call arranges all
of the following:

1. Declares a new subscribe queue with an unguessable name (see
   ``subscribe_key``).

2. Binds the subscribe queue to the ``event_key`` defined by the
   ``ILongPollEvent`` adapter of ``some_object``.

3. Puts the ``subscribe_key`` in the request's JSON cache as
   ``longpoll.key``, sets ``longpoll.uri`` so that the in-page
   JavaScript knows where to connect, and adds the ``subscribe_key``
   to the ``longpoll.subscriptions`` list.


Testing
=======

To test that your code triggers the appropriate long-poll events, see
`LongPollEventRecord`_ and the context manager
`capture_longpoll_emissions`_::

 from lp.services.longpoll.testing import (
     capture_longpoll_emissions,
     LongPollEventRecord,
     )

 with capture_longpoll_emissions() as log:
     pass
     # Here something should trigger a long poll event.

 expected = LongPollEventRecord(
     "my-event-key", {
         "event_key": "my-event-key",
         "something": "somedata"})

 self.assertEqual([expected], log)

.. _`LongPollEventRecord`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py

.. _`capture_longpoll_emissions`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py


Job done
========

That's the back-end plumbing, now it's up to the in-page JavaScript to
know what to do with the events that it'll be able to long-poll for...

Next: `/JavaScript`_

LongPoll (last edited 2011-10-04 20:06:01 by allenap)