RabbitMQ in Launchpad

This page describes how we use RabbitMQ in Launchpad and some of its deployment nuances.

This page is also currently WIP.

Crash course in RabbitMQ

Each RabbitMQ instance has one or more "virtual hosts" or vhosts. These are a convenient way to manage separate instances of rabbit via the same server instance - it could also be achieved with separate server instances, but you must set up at least one vhost on the instance.

RabbitMQ has queues, which have a producer and one or more consumers. The producer needs to set up:

• an "exchange"
• a "routing key"
• a "queue"

It then needs to bind the queue to zero or more pairs of exchange and routing key. When sending a message, it sends to the exchange and routing key combination, and the message is then available to consumers on all the queues that are bound to those.

See this producer example:

from amqplib import client_0_8 as amqp

conn = amqp.Connection(
    host='localhost:5672',
    userid='guest',
    password='guest',
    virtual_host='/',  # "/" is the default vhost.
    insist=False)
ch = conn.channel()
ch.exchange_declare(
    "XXX.notifications-exchange",
    "direct",
    durable=False,  # Whether the exchange persists or not.
    auto_delete=False)
ch.queue_declare("XXX.notifications-queue.foo")
ch.queue_bind(
    queue='XXX.notifications-queue.foo',
    exchange='XXX.notifications-exchange',
    routing_key="XXX.notifications-queue.foo")
msg = amqp.Message('{"boo": 1}')
ch.basic_publish(
    exchange="XXX.notifications-exchange",
    routing_key="XXX.notifications-queue.foo",
    msg=msg)

Long polling

See lp:txlongpoll for a project that describes a so-called long-polling server. It is Twisted-based and maps HTTP requests to consumption of messages on Rabbit queues.

This means we can have Javascript in the browser doing XHR to the long poll server and block on long-running jobs until the job emits a message saying it's done. There is currently work in progress to make merge proposal diffs appear as soon as the job completes so that the end user doesn't have to refresh the MP page.

Long poll request format

XXX (detail the URL; uuid, sequence etc)

Workflow

XXX