#format rst ========================= Long-polling in Launchpad ========================= .. contents:: Overview ======== In order that browsers may receive as-it-happens updates of server processes that run asynchronously with respect to the web application - of which there are many in Launchpad - we're using a combination of a message queue, RabbitMQ_, a AMQP to HTTP bridge, txlongpoll_, some additional abstraction/convenience APIs within Launchpad, and a JavaScript component that terminates the event pipeline in the browser. This document talks mostly about the APIs within Launchpad, but will also touch on RabbitMQ_, txlongpoll_, and the `JavaScript component`_. .. _RabbitMQ: http://www.rabbitmq.com/ .. _txlongpoll: https://launchpad.net/txlongpoll .. _`JavaScript component`: /JavaScript Here is an overview of all the components involved in long-polling in Launchpad:: +------------+ LP.cache.longpoll.{uri, key} | LP |<-----------(1)-----------------+ +------------+ | ^ v +-----------+ | +--------------------------+ | |<---LongPollEvent-(3)-+ | Browser | | RabbitMQ | |--------------------------| | Server | | | | |<---RabbitEvent---(4)-+ | longpolljs (JSEvent (6))| +-----------+ | +--------------------------+ v ^ +------------+ /{uri}?uuid={key}&sequence=0 | | txlongpoll |<------------(2)----------------+ +------------+ --(5)-> <-(7)-- The workflow for a typical usage of the long-poll system will go something like this: 1) The LP web server serves a web page with ``LP.cache.longpoll.uri`` and ``LP.cache.longpoll.key`` populated. If these are not populated, the long-poll machinery will stop here. 2) The JavaScript long-poll library then uses these to initiate a long lasting XHR request to the Twisted-based ``txlongpoll`` server. 3) Some time later, something noteworthy happens in Launchpad, and the code that wants to tell the world obtains an object providing ``ILongPollEvent``, typically via adaption. It then uses ``long_poll_event.emit(**payload)`` to send a message out via RabbitMQ, keyed with ``long_poll_event.event_key``. 4) The event will be routed by RabbitMQ out to listeners in ``txlongpoll``. 5) The long lasting XHR request to the ``txlongpoll`` server returns (with the JSONified ``payload`` that was passed to ``long_poll_event.emit(...)``). 6) The JavaScript long-poll library then propagates this payload via YUI's event system so that any piece of in-page code can receive events from server-side. 7) The JavaScript long-poll library now reconnects to ``txlongpoll`` to await additional events. Long-poll's usage ================= - `Merge proposal's diff update`_: merge proposal's diffs are updated via XHR as soon as new version is available (no more page reloads!) .. _`Merge proposal's diff update`: /MPDiffUpdate RabbitMQ ======== RabbitMQ_ is the messaging system used to connect events between discrete components. In this case it connects long running jobs, that emit an event when finishing, to the txlongpoll_ server which informs the browser. txlongpoll ========== `lp:txlongpoll`_ accepts HTTP requests from browsers and holds them open (for ~4-5 minutes at a time) while waiting for messages to arrive from the messaging system, to which it connects via AMQP. .. _lp:txlongpoll: https://launchpad.net/txlongpoll It is based on Twisted_ and is a completely separate service to Launchpad; it runs stand-alone and does not depend on Launchpad in any way. .. _Twisted: http://twistedmatrix.com/ Long-poll request format ======================== txlongpoll_ awaits HTTP GET requests of the form ``/uuid={queue_name}&sequence={sequence_id}`` The ``uuid`` specifies a queue that should already exist and be subscribed (bound) to events from an exchange within RabbitMQ_. The ``sequence`` is a sequential integer that txlongpoll_ requires and is used to better support load-balancing. It is incremented before each request. ---- Writing code on the server side =============================== LongPollEvent ------------- `LongPollEvent`_ is the abstract adapter base class that should be used to create custom event adapters. **Why?** -- *Julian* All that's needed is an implementation of ``ILongPollEvent``; subclassing ``LongPollEvent`` is not *required*. However, it has a sensible implementation of ``emit`` that is either usable as-is, or worth reusing via calling-up from subclasses. -- *Gavin* .. _LongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/event.py Sub-classes need to define the `event_key` property and declare something along the lines of:: class LongPollAwesomeThingEvent(LongPollEvent): adapts(IAwesomeThing) implements(ILongPollEvent) .. **What is event_key for?** -- *Julian* See `ILongPollEvent`_ ;) It is the key with which events will be emitted. It should be predictable and stable. This is the key to which queues can be subscribed. An example could be ``longpoll.event.branchmergeproposal.1234``. -- *Gavin* .. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py Alternatively, use the `long_poll_event` class decorator:: @long_poll_event(IAwesomeThing) class LongPollAwesomeThingEvent(LongPollEvent): ... In both cases the adapter should be registered in a `configure.zcml` somewhere sensible:: See `LongPollStormEvent`_ and its supporting code for a cargo-cultable example of this, and read on for an explanation of it. .. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py zope.event bridge for Storm life-cycle events --------------------------------------------- As a convenience, subscription handlers for `life-cycle events`_ of Storm model objects already exist. They generate long-poll events named ``longpoll.event.{table_name}.{object_primary_key}`` with the type of event ("created", "deleted" or "modified") in the event payload. .. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py For instance, if ``storm_object`` is an object from a table named ``FakeTable`` with a primary id of 1234, the following call:: notify(ObjectDeletedEvent(storm_object)) ... will result in an event being fired with the following data:: {"event_key": "longpoll.event.faketable.1234", "what": "deleted"}) ILongPollSubscriber ------------------- ``ILongPollSubscriber`` describes a subscriber to long-poll events. It defines a ``subscribe_key`` property: "The key which the subscriber must know in order to be able to long-poll for subscribed events. Should be infeasible to guess, a UUID for example." An example could be ``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``. It also defines a ``subscribe`` method that can be passed an object that provides ``ILongPollEvent``. In the back-end this ends up creating a queue named ``subscribe_key`` which is bound to event's ``event_key`` (not exclusively; it can be bound to multiple events). As the time of this writing only a request (as in ``IApplicationRequest``) can be used as a subscriber using LongPollApplicationRequestSubscriber_ **I don't understand what a "request" and a "subscriber" is in this context** -- *Julian* I think I've clarified that above. -- *Gavin* .. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py This adapter ensures that the request's JSON cache is stuffed with the necessary information that the JavaScript long-poll component needs to operate. A request can be subscribed to events quite simply:: ILongPollSubscriber(request).subscribe(FakeEvent()) However, there's an even easier way, using ``lp.app.longpoll.subscribe``:: subscribe(FakeEvent()) Or, when ``some_object`` can be adapted to ``ILongPollEvent`` this reduces even further:: subscribe(some_object) ILongPollSubscriber for Storm ----------------------------- If ``model_object`` is a Storm model object (i.e. is an instance of ``Storm``) then a simple ``subscribe(model_object)`` call arranges all of the following: 1. Declares a new subscribe queue with an unguessable name (see ``subscribe_key``). 2. Binds the subscribe queue to the ``event_key`` defined by the ``ILongPollEvent`` adapter of ``some_object``. 3. Puts the ``subscribe_key`` in the request's JSON cache as ``longpoll.key``, sets ``longpoll.uri`` so that the in-page JavaScript knows where to connect, and adds the ``subscribe_key`` to the ``longpoll.subscriptions`` list. Testing ======= To test that your code triggers the appropriate long-poll events, see `LongPollEventRecord`_ and the context manager `capture_longpoll_emissions`_:: from lp.services.longpoll.testing import ( capture_longpoll_emissions, LongPollEventRecord, ) with capture_longpoll_emissions() as log: pass # Here something should trigger a long poll event. expected = LongPollEventRecord( "my-event-key", { "event_key": "my-event-key", "something": "somedata"}) self.assertEqual([expected], log) .. _`LongPollEventRecord`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py .. _`capture_longpoll_emissions`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py Job done ======== That's the back-end plumbing, now it's up to the in-page JavaScript to know what to do with the events that it'll be able to long-poll for... Next: `/JavaScript`_