Size: 10039
Comment: reformat some parts
|
Size: 13061
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 33: | Line 33: |
1) The LP web server serves a web page with LP.cache.longpoll.uri and LP.cache.longpoll.key populated. (If these are not populated, the LongPoll machinery will stop here) 2) The javascript longpoll library then uses these to initiate a long lasting XHR request to the twisted based LongPoll server. 3) (Some time later) The LP app server uses LongPollEvent.emit(payload) to send a message to the RabbitMQ server (if the server is available) 4) The event will then be published by RabbitMQ to the txlongpoll server. 5) The long lasting XHR request to the txlongpoll server returns (with the JSONified payload from the LongPollEvent). 6) This is handled by the javascript longpoll library which triggers a javascript event with the payload. 7) The javascript longpoll library now reconnects to be able to handle other events. |
1) The LP web server serves a web page with LP.cache.longpoll.uri and LP.cache.longpoll.key populated. (If these are not populated, the LongPoll machinery will stop here) 2) The javascript longpoll library then uses these to initiate a long lasting XHR request to the twisted based LongPoll server. 3) (Some time later) The LP app server uses LongPollEvent.emit(payload) to send a message to the RabbitMQ server (if the server is available) 4) The event will then be published by RabbitMQ to the txlongpoll server. 5) The long lasting XHR request to the txlongpoll server returns (with the JSONified payload from the LongPollEvent). 6) This is handled by the javascript longpoll library which triggers a javascript event with the payload. 7) The javascript longpoll library now reconnects to be able to handle other events. LongPoll's usage ================ - `Merge proposal's diff update`_: merge proposal's diffs are updated via XHR as soon as new version is available (no more page reloads!) .. _`Merge proposal's diff update`: /MPDiffUpdate |
Line 44: | Line 61: |
RabbitMQ_ is the messaging system used to connect events between discrete components. In this case it connects long running jobs, that emit an event when finishing, to the "longpoll" server which informs the browser. | RabbitMQ_ is the messaging system used to connect events between discrete components. In this case it connects long running jobs, that emit an event when finishing, to the "longpoll" server which informs the browser. |
Line 51: | Line 71: |
See `lp:txlongpoll`_ for a project that describes a so-called long-polling server. It is Twisted-based and maps HTTP requests to consumption of messages on Rabbit queues. It is a completely separate service to Launchpad. | See `lp:txlongpoll`_ for a project that describes a so-called long-polling server. It is Twisted-based and maps HTTP requests to consumption of messages on Rabbit queues. It is a completely separate service to Launchpad. |
Line 55: | Line 78: |
This means we can have Javascript in the browser doing XHR to the LongPoll server and block on long-running jobs until the job emits a message saying it's done. There is currently work in progress to make merge proposal diffs appear as soon as the job completes so that the end user doesn't have to refresh the MP page. | This means we can have Javascript in the browser doing XHR to the LongPoll server and block on long-running jobs until the job emits a message saying it's done. There is currently work in progress to make merge proposal diffs appear as soon as the job completes so that the end user doesn't have to refresh the MP page. |
Line 60: | Line 87: |
The LongPoll server awaits connections of the form '/uuid={queue_name}&sequence={sequence_id}'. The uuid parameter will be used to connect to a specific queue. The sequence parameter is a sequential integer that RabbitMQ requires to fetch successive events from the same queue. **how is this sequence driven, exactly? Why is it required?** -- *Julian* |
The LongPoll server awaits connections of the form '/uuid={queue_name}&sequence={sequence_id}'. The uuid parameter will be used to connect to a specific queue. The sequence parameter is a sequential integer that RabbitMQ requires to fetch successive events from the same queue. **How is this sequence driven, exactly? Why is it required?** -- *Julian* It is required by ``txlongpoll`` and I think serves one main purpose: cache busting. -- *Gavin* |
Line 71: | Line 107: |
`LongPollEvent`_ is the abstract adapter base class that should be used to create custom event adapters. **why?** -- *Julian* |
`LongPollEvent`_ is the abstract adapter base class that should be used to create custom event adapters. **Why?** -- *Julian* All that's needed is an implementation of ``ILongPollEvent``; subclassing ``LongPollEvent`` is not *required*. However, it has a sensible implementation of ``emit`` that is either usable as-is, or worth reusing via calling-up from subclasses. -- *Gavin* |
Line 76: | Line 119: |
Sub-classes need to define the `event_key` property and declare something along the lines of:: | Sub-classes need to define the `event_key` property and declare something along the lines of:: |
Line 82: | Line 126: |
**what is event_key for?** -- *Julian* | .. **What is event_key for?** -- *Julian* See `ILongPollEvent`_ ;) It is the key with which events will be emitted. It should be predictable and stable. This is the key to which queues can be subscribed. An example could be ``longpoll.event.branchmergeproposal.1234``. -- *Gavin* .. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py |
Line 90: | Line 143: |
In both cases the adapter should be registered in a `configure.zcml` somewhere sensible:: | In both cases the adapter should be registered in a `configure.zcml` somewhere sensible:: |
Line 94: | Line 148: |
**EXAMPLE NEEDED HERE** -- *Julian* zope.event bridge ----------------- As a convenience, adapters for the storm events ObjectCreatedEvent, ObjectDeletedEvent and ObjectModifiedEvent already exist. They generate events named "longpoll.event.{table_name}.{object_primary_key}" with the type of event ("created", "deleted" or "modified") in the event payload. For instance, if storm_object is an object from a table named FakeTable with a primary id of 1234, calling:: |
.. **EXAMPLE NEEDED HERE** -- *Julian* See `LongPollStormEvent`_ -- *Gavin* .. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py zope.event bridge for Storm life-cycle events --------------------------------------------- As a convenience, subscription handlers for `life-cycle events`_ (IObjectCreatedEvent, IObjectDeletedEvent, IObjectModifiedEvent) of Storm model objects already exist. They generate long-poll events named "longpoll.event.{table_name}.{object_primary_key}" with the type of event ("created", "deleted" or "modified") in the event payload. .. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py For instance, if ``storm_object`` is an object from a table named ``FakeTable`` with a primary id of 1234, calling:: |
Line 113: | Line 180: |
To test that your code triggers the appropriate long poll events, use the class `LongPollEventRecord`_ and the context manager `capture_longpoll_emissions`_:: | To test that your code triggers the appropriate long poll events, use the class `LongPollEventRecord`_ and the context manager `capture_longpoll_emissions`_:: |
Line 129: | Line 198: |
self.assertEqual(expected, log) | self.assertEqual([expected], log) |
Line 140: | Line 209: |
As the time of this writing only a request can be used as a subscriber using LongPollApplicationRequestSubscriber_ **I don't understand what a "request" and a "subscriber" is in this context** -- *Julian* |
``ILongPollSubscriber`` describes a subscriber to long-poll events. It defines a ``subscribe_key`` property: "The key which the subscriber must know in order to be able to long-poll for subscribed events. Should be infeasible to guess, a UUID for example." An example could be ``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``. It also defines a ``subscribe`` method that can be passed an object that provides ``ILongPollEvent``. In the back-end this ends up creating a queue named ``subscribe_key`` which is bound to event's ``event_key`` (not exclusively; it can be bound to multiple events). As the time of this writing only a request (as in ``IApplicationRequest``) can be used as a subscriber using LongPollApplicationRequestSubscriber_ **I don't understand what a "request" and a "subscriber" is in this context** -- *Julian* I think I've clarified that above. -- *Gavin* |
Line 146: | Line 235: |
Once a LongPollApplicationRequestSubscriber object is created, individual events can be subscribed to it:: subscriber = LongPollApplicationRequestSubscriber(request) subscriber.subscribe(FakeEvent()) After the first event has been subscribed, LP.cache.longpoll.key will be populated with a key (identifying the RabbitMQ queue) specific to this request. **I don't understand how this stuff fits in and where it is used, can you explain with an example?** -- *Julian* |
This adapter ensures that the request's JSON cache is stuffed with the necessary information for the JavaScript component of LongPoll can operate. A request can be subscribed to a events quite simply:: ILongPollSubscriber(request).subscribe(FakeEvent()) However, there's an even easier way, using ``lp.app.longpoll.subscribe``:: subscribe(FakeEvent()) Or, when ``some_object`` can be adapted to ``ILongPollEvent``:: subscribe(some_object) For example, if ``some_object`` is a Storm model object (i.e. inherits from Storm, or indirectly via SQLBase) then this single call arranges the following: 1. Declares a new subscribe queue with an unguessable name (see ``subscribe_key``). 2. Binds the subscribe queue to the ``event_key`` defined by the ``ILongPollEvent`` adapter of ``some_object``. 3. Puts the ``subscribe_key`` in the request's JSON cache as ``longpoll.key``, sets ``longpoll.uri`` so that the in-page JavaScript knows where to connect, and adds the ``subscribe_key`` to the ``longpoll.subscriptions`` list. That's the back-end plumbing, now it's up to the in-page JavaScript to know what to do with the events that it'll be able to long-poll for... |
Line 160: | Line 275: |
The `long poll Javascript library`_ is responsible for connecting to the LongPoll server and firing Javascript events to reflect the state of this connection and make RabbitMQ events available to the Javascript layer. | The `long poll Javascript library`_ is responsible for connecting to the LongPoll server and firing Javascript events to reflect the state of this connection and make RabbitMQ events available to the Javascript layer. |
Line 167: | Line 285: |
The long poll library will issue a connection to the long poll server iff LP.cache.longpoll.uri and LP.cache.longpoll.key are populated in the JSON cache. After each successful or failed connection, the long poll library reconnects to the long poll server to be able to handle other events. |
The long poll library will issue a connection to the long poll server iff LP.cache.longpoll.uri and LP.cache.longpoll.key are populated in the JSON cache. After each successful or failed connection, the long poll library reconnects to the long poll server to be able to handle other events. |
Line 173: | Line 294: |
Two kind of events are fired by the long poll Javascript library. Events deriving from LongPollEvents and "system" events indicating the state of the long polling connection. Here is a list of all the possible events fired by the Javascript LongPoll library: | Two kind of events are fired by the long poll Javascript library. Events deriving from LongPollEvents and "system" events indicating the state of the long polling connection. Here is a list of all the possible events fired by the Javascript LongPoll library: |
Line 176: | Line 300: |
fired each time RabbitMQ passes an event created using the LongPollEvent adapter (event_key being the name of the RabbitMQ event). | fired each time RabbitMQ passes an event created using the LongPollEvent adapter (event_key being the name of the RabbitMQ event). |
Line 198: | Line 324: |
fired each time a long poll transaction fails. This might be because of connection problems or because the library failed to parse the event payload. To avoid hammering the server in case of an outage, the Long poll Javascript library waits for 1 second before reconnecting. Also, after 5 failed connections, the library will wait 3 minutes before trying to reconnect. The following events can be used to make the end user aware of the connection status. |
fired each time a long poll transaction fails. This might be because of connection problems or because the library failed to parse the event payload. To avoid hammering the server in case of an outage, the Long poll Javascript library waits for 1 second before reconnecting. Also, after 5 failed connections, the library will wait 3 minutes before trying to reconnect. The following events can be used to make the end user aware of the connection status. |
Line 208: | Line 336: |
fired each time the library switches in 'longdelay' mode (i.e. after 5 failed connections). | fired each time the library switches in 'longdelay' mode (i.e. after 5 failed connections). |
Line 212: | Line 341: |
fired each time the library switches back in 'shortdelay' mode (i.e. after at least 5 failed connections and then after a successful connection). |
fired each time the library switches back in 'shortdelay' mode (i.e. after at least 5 failed connections and then after a successful connection). |
Line 218: | Line 348: |
Since all the long poll Javascript library will be doing is firing Javascript events, testing a Javascript application that interacts with this library is pretty straightforward. Simply fire the events and make sure the application reacts appropriately:: | Since all the long poll Javascript library will be doing is firing Javascript events, testing a Javascript application that interacts with this library is pretty straightforward. Simply fire the events and make sure the application reacts appropriately:: |
======== LongPoll ======== .. contents:: Overview ======== Here is an overview of all the components involved in long polling:: +------------+ LP.cache.longpoll.{uri, key} | LP |<-----------(1)-----------------+ +------------+ | ^ v +-----------+ | +--------------------------+ | |<---LongPollEvent-(3)-+ | Browser | | RabbitMQ | |--------------------------| | Server | | | | |<---RabbitEvent---(4)-+ | longpolljs (JSEvent (6))| +-----------+ | +--------------------------+ v ^ +------------+ /{uri}?uuid={key}&sequence=0 | | txlongpoll |<------------(2)----------------+ +------------+ --(5)-> <-(7)-- The workflow for a typical usage of the LongPoll system of LP will go like this: 1) The LP web server serves a web page with LP.cache.longpoll.uri and LP.cache.longpoll.key populated. (If these are not populated, the LongPoll machinery will stop here) 2) The javascript longpoll library then uses these to initiate a long lasting XHR request to the twisted based LongPoll server. 3) (Some time later) The LP app server uses LongPollEvent.emit(payload) to send a message to the RabbitMQ server (if the server is available) 4) The event will then be published by RabbitMQ to the txlongpoll server. 5) The long lasting XHR request to the txlongpoll server returns (with the JSONified payload from the LongPollEvent). 6) This is handled by the javascript longpoll library which triggers a javascript event with the payload. 7) The javascript longpoll library now reconnects to be able to handle other events. LongPoll's usage ================ - `Merge proposal's diff update`_: merge proposal's diffs are updated via XHR as soon as new version is available (no more page reloads!) .. _`Merge proposal's diff update`: /MPDiffUpdate RabbitMQ ======== RabbitMQ_ is the messaging system used to connect events between discrete components. In this case it connects long running jobs, that emit an event when finishing, to the "longpoll" server which informs the browser. .. _RabbitMQ: RabbitMQ LongPoll server =============== See `lp:txlongpoll`_ for a project that describes a so-called long-polling server. It is Twisted-based and maps HTTP requests to consumption of messages on Rabbit queues. It is a completely separate service to Launchpad. .. _lp:txlongpoll: https://launchpad.net/txlongpoll This means we can have Javascript in the browser doing XHR to the LongPoll server and block on long-running jobs until the job emits a message saying it's done. There is currently work in progress to make merge proposal diffs appear as soon as the job completes so that the end user doesn't have to refresh the MP page. LongPoll request format ======================= The LongPoll server awaits connections of the form '/uuid={queue_name}&sequence={sequence_id}'. The uuid parameter will be used to connect to a specific queue. The sequence parameter is a sequential integer that RabbitMQ requires to fetch successive events from the same queue. **How is this sequence driven, exactly? Why is it required?** -- *Julian* It is required by ``txlongpoll`` and I think serves one main purpose: cache busting. -- *Gavin* ---- Writing code on the server side =============================== LongPollEvent class ------------------- `LongPollEvent`_ is the abstract adapter base class that should be used to create custom event adapters. **Why?** -- *Julian* All that's needed is an implementation of ``ILongPollEvent``; subclassing ``LongPollEvent`` is not *required*. However, it has a sensible implementation of ``emit`` that is either usable as-is, or worth reusing via calling-up from subclasses. -- *Gavin* .. _LongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/event.py Sub-classes need to define the `event_key` property and declare something along the lines of:: class LongPollAwesomeThingEvent(LongPollEvent): adapts(IAwesomeThing) implements(ILongPollEvent) .. **What is event_key for?** -- *Julian* See `ILongPollEvent`_ ;) It is the key with which events will be emitted. It should be predictable and stable. This is the key to which queues can be subscribed. An example could be ``longpoll.event.branchmergeproposal.1234``. -- *Gavin* .. _ILongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/interfaces.py Alternatively, use the `long_poll_event` class decorator:: @long_poll_event(IAwesomeThing) class LongPollAwesomeThingEvent(LongPollEvent): ... In both cases the adapter should be registered in a `configure.zcml` somewhere sensible:: <adapter factory=".adapters.LongPollAwesomeThingEvent" /> .. **EXAMPLE NEEDED HERE** -- *Julian* See `LongPollStormEvent`_ -- *Gavin* .. _LongPollStormEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/storm.py zope.event bridge for Storm life-cycle events --------------------------------------------- As a convenience, subscription handlers for `life-cycle events`_ (IObjectCreatedEvent, IObjectDeletedEvent, IObjectModifiedEvent) of Storm model objects already exist. They generate long-poll events named "longpoll.event.{table_name}.{object_primary_key}" with the type of event ("created", "deleted" or "modified") in the event payload. .. _life-cycle events: http://bazaar.launchpad.net/~lazr-developers/lazr.lifecycle/trunk/view/head:/src/lazr/lifecycle/interfaces.py For instance, if ``storm_object`` is an object from a table named ``FakeTable`` with a primary id of 1234, calling:: notify(ObjectDeletedEvent(storm_object)) ... will result in an event being fired with the following data:: {"event_key": "longpoll.event.faketable.1234", "what": "deleted"}) Testing ------- To test that your code triggers the appropriate long poll events, use the class `LongPollEventRecord`_ and the context manager `capture_longpoll_emissions`_:: from lp.services.longpoll.testing import ( capture_longpoll_emissions, LongPollEventRecord, ) with capture_longpoll_emissions() as log: pass # Here something should trigger a long poll event. expected = LongPollEventRecord( "my-event-key", { "event_key": "my-event-key", "something": "somedata"}) self.assertEqual([expected], log) .. _`LongPollEventRecord`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py .. _`capture_longpoll_emissions`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/testing.py ---- LongPoll subscription ===================== ``ILongPollSubscriber`` describes a subscriber to long-poll events. It defines a ``subscribe_key`` property: "The key which the subscriber must know in order to be able to long-poll for subscribed events. Should be infeasible to guess, a UUID for example." An example could be ``longpoll.subscribe.a966e051-8817-4897-840f-9de18404c8f0``. It also defines a ``subscribe`` method that can be passed an object that provides ``ILongPollEvent``. In the back-end this ends up creating a queue named ``subscribe_key`` which is bound to event's ``event_key`` (not exclusively; it can be bound to multiple events). As the time of this writing only a request (as in ``IApplicationRequest``) can be used as a subscriber using LongPollApplicationRequestSubscriber_ **I don't understand what a "request" and a "subscriber" is in this context** -- *Julian* I think I've clarified that above. -- *Gavin* .. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py This adapter ensures that the request's JSON cache is stuffed with the necessary information for the JavaScript component of LongPoll can operate. A request can be subscribed to a events quite simply:: ILongPollSubscriber(request).subscribe(FakeEvent()) However, there's an even easier way, using ``lp.app.longpoll.subscribe``:: subscribe(FakeEvent()) Or, when ``some_object`` can be adapted to ``ILongPollEvent``:: subscribe(some_object) For example, if ``some_object`` is a Storm model object (i.e. inherits from Storm, or indirectly via SQLBase) then this single call arranges the following: 1. Declares a new subscribe queue with an unguessable name (see ``subscribe_key``). 2. Binds the subscribe queue to the ``event_key`` defined by the ``ILongPollEvent`` adapter of ``some_object``. 3. Puts the ``subscribe_key`` in the request's JSON cache as ``longpoll.key``, sets ``longpoll.uri`` so that the in-page JavaScript knows where to connect, and adds the ``subscribe_key`` to the ``longpoll.subscriptions`` list. That's the back-end plumbing, now it's up to the in-page JavaScript to know what to do with the events that it'll be able to long-poll for... ---- LongPoll Javascript library =========================== The `long poll Javascript library`_ is responsible for connecting to the LongPoll server and firing Javascript events to reflect the state of this connection and make RabbitMQ events available to the Javascript layer. .. _`long poll Javascript library`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/app/longpoll/javascript/longpoll.js Connection ---------- The long poll library will issue a connection to the long poll server iff LP.cache.longpoll.uri and LP.cache.longpoll.key are populated in the JSON cache. After each successful or failed connection, the long poll library reconnects to the long poll server to be able to handle other events. Events ------ Two kind of events are fired by the long poll Javascript library. Events deriving from LongPollEvents and "system" events indicating the state of the long polling connection. Here is a list of all the possible events fired by the Javascript LongPoll library: *event_key* fired each time RabbitMQ passes an event created using the LongPollEvent adapter (event_key being the name of the RabbitMQ event). For instance, this in LP python code:: # subscription code omitted. class FakeEvent(LongPollEvent): implements(ILongPollEvent) @property def event_key(self): return "event-key-test" event = FakeEvent("source") event_data = {"hello": 1234} event.emit(**event_data) ... will result in an event being fired on the Javascript side:: Y.fire("event-key-test", {"hello": 1234}); *'lp.app.longpoll.failure'* fired each time a long poll transaction fails. This might be because of connection problems or because the library failed to parse the event payload. To avoid hammering the server in case of an outage, the Long poll Javascript library waits for 1 second before reconnecting. Also, after 5 failed connections, the library will wait 3 minutes before trying to reconnect. The following events can be used to make the end user aware of the connection status. *'lp.app.longpoll.longdelay'* fired each time the library switches in 'longdelay' mode (i.e. after 5 failed connections). *'lp.app.longpoll.shortdelay'* fired each time the library switches back in 'shortdelay' mode (i.e. after at least 5 failed connections and then after a successful connection). Testing ------- Since all the long poll Javascript library will be doing is firing Javascript events, testing a Javascript application that interacts with this library is pretty straightforward. Simply fire the events and make sure the application reacts appropriately:: // Fire "event-key-test". Y.fire("event-key-test", {"hello": 1234}); // Simulate a failure. Y.fire("lp.app.longpoll.failure"); // Simulate the library going into 'longdelay' mode. Y.fire("lp.app.longpoll.longdelay");