=====================
RabbitMQ in Launchpad
=====================

This page describes how we use RabbitMQ in Launchpad and some of its deployment nuances.

This page is also currently WIP.

Crash course in RabbitMQ
========================

Each RabbitMQ instance has one or more "virtual hosts" or vhosts. These are a convenient way to manage separate instances of rabbit via the same server instance - it could also be achieved with separate server instances, but you must set up at least one vhost on the instance.

RabbitMQ has queues, which have a producer and one or more consumers. The producer needs to set up:
 * an "exchange"
 * a "routing key"
 * a "queue"

It then needs to bind the queue to zero or more pairs of exchange and routing key.  When sending a message, it sends to the exchange and routing key combination, and the message is then available to consumers on all the queues that are bound to those.

See this producer example::

 from amqplib import client_0_8 as amqp
 
 conn = amqp.Connection(
     host='localhost:5672',
     userid='guest',
     password='guest',
     virtual_host='/',  # "/" is the default vhost.
     insist=False)
 ch = conn.channel()
 ch.exchange_declare(
     "XXX.notifications-exchange",
     "direct",
     durable=False,  # Whether the exchange persists or not.
     auto_delete=False)
 ch.queue_declare("XXX.notifications-queue.foo")
 ch.queue_bind(
     queue='XXX.notifications-queue.foo',
     exchange='XXX.notifications-exchange',
     routing_key="XXX.notifications-queue.foo")
 msg = amqp.Message('{"boo": 1}')
 ch.basic_publish(
     exchange="XXX.notifications-exchange",
     routing_key="XXX.notifications-queue.foo",
     msg=msg)

Management Interface
====================

There is a plugin that is packaged as ``rabbitmq-management`` in `William Grant's PPA`_ and it provides a JSON API and a web interface to managing the running Rabbit server.  Once it's install, access it by browsing to ``http://localhost:55672/``. The default user/pass is guest/guest.

.. _William Grant's PPA: http://launchpad.net/~wgrant/+archive/rabbitmq

Here you can set up another vhost separate to the default one of "/". For development purposes, set up one called "launchpad" and be sure to set permissions for the guest user to access it. 


Long polling
============

Overview
--------

Here is an overview of all the components involved in long polling::


                              +------------+  LP.cache.longpoll.{uri, key}
                              |     LP     |<-----------(1)-----------------+
                              +------------+                                |
                                    ^                                       v
 +-----------+                      |                           +--------------------------+
 |           |<---LongPollEvent-(3)-+                           |        Browser           |
 | RabbitMQ  |                                                  |--------------------------|
 |  Server   |                                                  |                          |
 |           |<---RabbitEvent---(4)-+                           | longpolljs  (JSEvent (6))|
 +-----------+                      |                           +--------------------------+                   
                                    v                                       ^
                              +------------+     /{uri}?uuid={key}&sequence |
                              | txlongpoll |<------------(2)----------------+
                              +------------+           --(5)->
                                                       <-(7)--

The workflow for a typical usage of the long poll system of LP will go like this:

1) The LP web server serves a web page with LP.cache.longpoll.uri and LP.cache.longpoll.key populated. (If these are not populated)
2) The javascript longpoll library then uses these to initiate a long lasting XHR request to the twisted based long poll server.
3) (Some time later) The LP app server uses LongPollEvent.emit(payload) to send a message to the RabbitMQ server (if the server is available)
4) The event will then be published by RabbitMQ to the txlongpoll server.
5) The long lasting XHR request to the txlongpoll server will now return (will the JSONified payload from the LongPollEvent).
6) This will be handled by the javascript longpoll library which will trigger a javascript event with the payload.
7) The javascript longpoll library will now reconnect to be able to handle other events.

Long poll server
----------------

See `lp:txlongpoll`_ for a project that describes a so-called long-polling server. It is Twisted-based and maps HTTP requests to consumption of messages on Rabbit queues.

.. _lp:txlongpoll: https://launchpad.net/txlongpoll

This means we can have Javascript in the browser doing XHR to the long poll server and block on long-running jobs until the job emits a message saying it's done. There is currently work in progress to make merge proposal diffs appear as soon as the job completes so that the end user doesn't have to refresh the MP page.

Long poll request format
------------------------

The Long poll server awaits connections of the form '/uuid={queue_name}&sequence={sequence_id}'. The uuid parameter will be used to connect to a specific queue. The sequence parameter is a sequential integer that RabbitMQ requires to fetch successive events on the same queue.

Long poll events
----------------

LongPollEvent
~~~~~~~~~~~~~

`LongPollEvent`_ is the abstract adapter base class that should be used to create a custom event adapters.

.. _LongPollEvent: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/event.py

Sub-classes need to define the `event_key` property and declare something along the lines of::

 class LongPollAwesomeThingEvent(LongPollEvent):
     adapts(IAwesomeThing)
     implements(ILongPollEvent)

Alternatively, use the `long_poll_event` class decorator::

 @long_poll_event(IAwesomeThing)
 class LongPollAwesomeThingEvent(LongPollEvent):
     ...

In both cases the adapter should be registered in a `configure.zcml` somewhere sensible::

 <adapter factory=".adapters.LongPollAwesomeThingEvent" />

zope.event bridge
~~~~~~~~~~~~~~~~~

As a convenience, adapters for the storm events ObjectCreatedEvent, ObjectDeletedEvent and ObjectModifiedEvent already exist. They generate events named "longpoll.event.{table_name}.{object_primary_key}" with the type of event ("created", "deleted" or "modified") in the payload.

For instance calling, if storm_object is an object from a table named FakeTable with a primary id of 1234::

 notify(ObjectDeletedEvent(storm_object))

Will result in this event being fired::

 {"event_key": "longpoll.event.faketable.1234",
  "what": "deleted"})

Long poll subscription
----------------------

As the time of this writing only a request can be used as a subscriber using LongPollApplicationRequestSubscriber_

.. _LongPollApplicationRequestSubscriber: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/services/longpoll/adapters/subscriber.py

Once a LongPollApplicationRequestSubscriber object is created, individual events can be subscribed to it::

 subscriber = LongPollApplicationRequestSubscriber(request)
 subscriber.subscribe(FakeEvent())

After the first event has been subscribed, LP.cache.longpoll.key will be populated with a key (identifying the RabbitMQ queue) specific to this request.

Long poll Javascript library
----------------------------

The `long poll Javascript library`_ is responsible for connecting to the long poll server and firing Javascript events to reflect the state of this connection and make RabbitMQ event available to the Javascript layer.

.. _`long poll Javascript library`: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/view/head:/lib/lp/app/longpoll/javascript/longpoll.js

Connection
~~~~~~~~~~

The long poll library will issue a connection to the long poll server iff LP.cache.longpoll.uri and LP.cache.longpoll.key are populated in the JSON cache.
After each successful or failed connection, the long poll library reconnects to the long poll server to be able to handle other events.

Events
~~~~~~

Two kind of events are fired by the long poll Javascript library.  Events deriving from LongPollEvents and "system" events indicating the state of the long polling transaction.

- event_key: fired each time RabbitMQ passed an event created using the LongPollEvent adapter.

For instance, this in LP python code::

 # subscription code omitted.
 class FakeEvent(LongPollEvent):

     implements(ILongPollEvent)

     @property
     def event_key(self):
        return "event-key-test"

 event = FakeEvent("source")
 event_data = {"hello": 1234}
 event.emit(**event_data)

Will result in an event being fired on the Javascript side::

 Y.fire("event-key-test", {"hello": 1234});

- 'lp.app.longpoll.failure': fired each time a long poll transaction fails. This might be because of connection problems or because the event payload failed to parse.

To avoid hammering the server in case of an outage, the Long poll Javascript library waits for 1 second before reconnecting. Also, after 5 failed connections, the library will wait 3 minutes before trying to reconnect:

- 'lp.app.longpoll.longdelay': fired each time the library switches in 'longdelay' mode (i.e. after 5 failed connections).

- 'lp.app.longpoll.shortdelay': fired each time the library switches back in 'shortdelay' mode (i.e. after at least 5 failed connections and then after a successful connection).