= Implementation Notes =

These are very rough so far.

The big picture is that we will have a bounce processor service that is responsible for listening and responding to two or three different channels (LMTP, RabbitMQ, and some HTTP protocol); and some associated changes to Launchpad.

I want to keep a given state in only one location. The bounce processor will be responsible for remembering the "score" of all bounced emails, and all associated data for maintaining the score.  Launchpad will only be responsible for remembering whether an account is in the "disabled preferred email" state.

== Bounce processor service ==

 * Service listens as an LMTP server (the stdlib has one in smtpd based on asyncore; Twisted [[http://twistedmatrix.com/pipermail/twisted-python/2006-April/012946.html|does not]] [[http://comments.gmane.org/gmane.comp.python.twisted/22306|currently have one]], [[http://www.ietf.org/rfc/rfc2033.txt|though it might be "easy"]]) for bounce messages forwarded from Canonical SMTP server. The LMTP server accepts all messages, pushes them into RabbitMQ to be handled, and returns OK.

 * Service listens to RabbitMQ.
   * bounce messages from LMTP: 
     * flufl.bounce identifies bounces, and gives us the affected email addresses for each bounce.  If message is a "permanent" bounce...
     * for each affected email address, if bounce score has not increased in the past 24 hours...
     * increase the score by 1.
     * cancel any scheduled bounce score reset calls for this email address
     * If score moves above disabling threshold for a given address...
       * communicate to Launchpad (existing XMLRPC channel, extended) that address has disabled status, and 
       * schedule a "poll" email for one week later.
     * else if not yet above disabling threshold...
       * schedule a bounce score reset call for seven days later.  This will set the bounce score to zero for this address unless it is cancelled.
   * smtp error messages from Launchpad:
     * If the error status is 450 or 5xx...
     * behave exactly as for affected email addresses in LMTP story.
   * "reset bounce score for this email address" message from Launchpad. XXX does this need to be synchronous instead, so that UIs can count on the change happening quickly and reliably after a page refresh?
     * set the bounce score for this address to zero
     * If disabled "poll" emails are scheduled, cancel them.
     * If "bounce score reset" calls are scheduled, cancel them.

 * Service listens to scheduled tasks. XXX what mechanism? RabbitMQ does not provide.  https://github.com/kr/beanstalkd/wiki/FAQ is recommended but rolling our own may be easier then getting another tool vetted and deployed.
  * "poll" emails once a week for four weeks after an email is disabled.  These need to be able to be cancelled.
  * "bounce score reset" to set an address' bounce score to zero unless the score is over the disable threshold.  These need to be able to be cancelled.

 * Service listens via HTTP to Zope or browser, TBD. XXX If we make it simply proxied via Apache, that means that the server can be used to verify the existence of email addresses if they have bounced in the past seven days or are disabled.  Is this OK for security/privacy?  If instead Zope has to be the go-between in order to make sure that the request only includes emails that the user should know about, a Zope thread will be consumed with waiting for the bounce server to reply over the network *or* we figure out a way to pass a socket back to asyncore, which might be fine; or we just give an iterable that returns nothing until it is ready (this will mean content-length should not be set...this may not work so well with the Apache proxy, which usually keeps the connection open to the client; we'd have to experiment.)
   * Given a list of email addresses, tell me the bounce scores and whether they are considered to be disabled (and maybe the full text of the most recent bounce email).

[Notes from the weeds:
Long term persistence would therefore need to keep track of these things.
- each email address would have a bounce score, a reference to the last bounce message (so we can get the date from the bounce information at least, and maybe also give visibility to the full message for customer help situations), last probe timestamp, disabled_status.
- bounce messages would minimally want dates, and message text.

As noted above, I believe all of the data stored could be eventually consistent.  Because of this, I'm interested in Cassandra as a persistence layer.]

== Launchpad ==

 * Launchpad listens on the existing XMLRPC channel for messages from the bounce processor that an email has moved to the "disabled" status.  If that email is someone's preferred email, make the account move into a "disabled primary email" status and send an email to all registered & validated emails for that user (see LEP).

 * Launchpad changes sending emails in three ways:

   * No email should be sent to a user with an account with the "disabled primary email" status.  This includes mailman AFAIK, which might be interesting.
   * ''All'' emails sent from Launchpad should include VERP information (e.g., "from = 'foo-bounces+gary.poster=canonical.com@launchpad.net'; smtplib.sendmail(from, to, msg)").  Including mailman in this list may be challenging.
   * When sending an email if the SMTP server returns a failure reply code, tell the bounce processor service over RabbitMQ, as described in the service description.

 * If a user has the "bounce score reset" status, show a banner on every page warning the user.  See LEP.

 * The email web page for users is modified to view and remove the "disabled primary email" status. Doing this sends a message to the bounce processor over RabbitMQ to reset the bounce score to zero, as described in the service description.

== Proposed incremental deliverables ==

XXX LP only stuff first, admin controlled.  ugly first, then pretty. XXX more