Bug Notifications and Subscriptions: Less Noise, More Control

We should make subscribing to bugs something people want to do, not dread.

Contact: GaryPoster and the Yellow squad
Originally Drafter: Deryck Hodge <deryck.hodge@canonical.com>
On Launchpad: The story-better-bug-notification tag.

As an Ubuntu developer
I want control over my Launchpad subscriptions
so that I can have an easier time following exactly what I care about in Launchpad.

As an upstream developer
I want control over what information I get from Launchpad
so that I can easily find bugs related to my software.

We are taking two primary approaches to these goals.

• Forward-looking action: Provide more control over subscriptions, so you get fewer unwanted notifications.

• Experience-based reaction: Provide a way to understand why you got a particular notification, and a way to react to it so you don't get notifications like it again.

In addition, we will make a small effort to do better batching of notifications.

If people get fewer Launchpad notifications that do not interest them, they will have an easier time finding and following the bugs and other events that they do care about.

Rationale

Why now?

This story was taken up after consultation with Brian Murray, representing Ubuntu Platform QA. Launchpad's current focus is Getting Bugs Off Ubuntu. As a feature slot came open for development, we asked Brian what would help Ubuntu best manage the amount of bugs they have. He said, better bug subscribing and notifications.

Brian says, "people subscribe mostly to packages, but still get too much mail"

Stats from DB confirm packages are the most subscribed to: https://pastebin.canonical.com/29229/

What value does this bring?

Fixing bug notifications to be less noisy and bug subscribing to allow finer grained controls will allow people to manage the high volume of bugs against packages. This brings value to both Ubuntu and upstream developers by allowing people to better find and follow bugs that interest them.

Stakeholders

Who will care about this work?

In particular, Ubuntu developers who are subscribed to large numbers of packages. In general, anyone who receives a lot of bug mail.

• bdmurray
• bryce
• leanne
• cjwatson
• pitti
• seb128
• ubuntu server team
• ubuntu kernel team
• keybuk (Note: too much bug mail became a serious problem to the end of 10.04 LTS)
• Elliot Murphy (or someone on his team)

When did we last talk to these stakeholders?

Only bdmurray has been consulted so far, and only in preparation for this story.

We hope to use iterative user interface testing to get early input into the work from selected stakeholders, and alpha testing for later input. The alpha testing group is malone-alpha.

Constraints and Requirements

Must

• Users can subscribe to individual bugs, choosing to filter by events. They can receive all notifications as they do now ("the bug receives any change"), or to only receive notifications if the "bug is opened, closed, or reopened," or to only receive notifications if the "bug is changed in any way other than a simple comment." Note that we may want to expand the most minimal filter, "bug is opened, closed, or reopened," to include moving to "incomplete," so that people filing bugs can see when more information is requested. (See further discussion on event filtering in "Nice to have.")

• Users can subscribe to notifications on the bugs of projects, packages, and distributions ("structural subscriptions") with any of the following filters, intersected. 674422

  • Only subscribe to certain events that trigger a notification: choose one among these options "bug is opened, closed, or reopened", "the bug is changed in any way other than a simple comment", or "the bug receives any change." Note that we may want to expand the most minimal filter, "bug is opened, closed, or reopened," to include moving to "incomplete," so that people filing bugs can see when more information is requested. (See further discussion on event filtering in "Nice to have.")

  • Users can filter out notifications generated because of a change they made themselves. This will be a global choice for each user. 548

  • Only subscribe to notifications that happen on bugs that are tagged with any of one or more tags or all of one or more tags. Users can include negative assertions about tags--so, for instance, you can subscribe to bugs that have one tag and do not have another. (See further discussion on this in "Nice to have.") 151129

  • Only subscribe to notifications that happen on bugs that have any of one or more selected statuses.

  • Only subscribe to notifications that happen on bugs that have any of one or more selected importances.

• Users can make multiple bug notification subscriptions for the same project, package, or distribution. Each can have different fliters as described above. Multiple subscriptions that overlap result in only a single notification. This means that, for instance, you can subscribe to all bug notifications in the Launchpad project that have the UI tag, and also bug notifications in the Launchpad project that show when a bug is opened or closed with the tag "bugjam2011". If a notification matches both subscriptions, you only get one copy of the notification.

• Users can edit and delete these subscriptions.

• Users can "mute" their notifications on individual bugs, so even if they would normally get a notification for it because of a subscription on a package, for instance, they will not hear notifications about that particular bug. 204980

• Each email notification will include a link to a page with all subscriptions that led to the receipt of that email. You can mute notifications from that particular bug, or edit or delete the subscriptions that made you receive the email. 649252

• Notifications will batch attachments that are added to the same bug by the same person within a certain amount of time (five minutes now, AIUI). 424849

• Email notifications include a X-Launchpad-Subscription header with the (optional, user-defined) names of the subscriptions that caused the notification to be sent. They also include the descriptions names in the email body, so that they can be used for filtering by mail clients such as Gmail that do not expose arbitrary headers for filtering.

Must not

• Actions that are done and then quickly undone will not generate notification emails 164196

Nice to have (Maybe)

• We provide a single, central page for a user to see and manage all of their subscriptions. This might be limited to bug subscriptions, or limited further to structural bug subscriptions.

• We provide a single, central page for a team to see and manage all of the related subscriptions.

• The page reached from the 'unsubscribe' link and/or the central user pages should also give you the option to turn off all your launchpad [bug?] mail.

• To add extra flexibility to the structural subscriptions filters described above in the "Must" section, the subscriptions allow the following approaches.

  • Users can choose to also receive notifications when a bug no longer matches the filter's description. For instance, if you are subscribed to a tag named "user-experience" because that is your domain in a given package, you may want to know when a bug loses that tag. (Note that this may cause confusion if you have made this gesture and yet are not subscribed to metadata events, because then you will not receive the notification you expect.) (Implementation note: after discussion with Deryck, sending notifications on a bug's diff like this sounds doable but possibly tricky.)

  • Finer-grained event filtering (please note that the behavior described in the "Must" section is already mostly implemented, via an enumeration.). The following two options are mutually exclusive.

    • Option 1. (Extend the existing enumeration.) In addition to being able to choose to receive events as described in "Must" (no event filtering, or "only when a bug is opened, closed, or reopened", or "if the bug is changed in any way other than a comment alone"), include two other options: "only when a bug is first opened" and "only when a bug is opened or the status changes". As before, users must choose only one of the full set of available event filters; they cannot be combined. The first new one is the most minimal optional; the second one is a bit more inclusive than "only when a bug is opened, closed, or reopened" but still does not trigger notifications for changes to other metadata, like importance changes and linked branches, or for comments.

    • Option 2. (Replace the enumeration with individually selectable choices.) Users can subscribe to fine-grained changes, like choosing explicitly to include or omit changes to each of initial creation, status, importance, linked branches, and comments. Please note the discussion about this feature in the "Thoughts?" section, titled "Unlimited configuration for notifications?"

  • Tag filtering:

    • Users can subscribe to events that happen on bugs that have no tags (this is partially implemented).

    • Users can subscribe to events that happen on bugs that have any tags (this is partially implemented).

    • Instead of a single broad "any" or "all" choice on the tag filter as described in the "Must" section, users can combine filter specifications with parentheses and boolean ANDs and ORs.

• The extra filtering described above for events is also available for direct subscriptions to bugs. This would almost certainly need to involve UI changes for the direct subscriptions.

• Users can have a gesture to "mute" a direct subscription. The only reason this would make sense, as opposed to simply deleting the direct subscription, would be because we (ab)use subscriptions to provide security access to private bugs. You could mute a direct subscription in order to keep access to the bug while stopping notifications from that bug. A better solution would be to change how we manage the security of private bugs, which I believe is something that Curtis Hovey is leading.

• Email notifications include options for nicer Gmail integration. One option is to ask users (globally, I'd assume) if they want us to insert their descriptions into their email address in a Gmail-friendly way. For instance, if a notification was sent to user@example.com because of two subscriptions with the descriptions of "bugs in foo" and "bugs in bar", respectively, it might result in a sending address of user+bugs-in-foo+bugs-in-bar@example.com. Using List-ID is another option, but there is an implementation concern: List-ID allows one value, and each of our notifications may have multiple sources. Also, some have raised concerns that we are not list-like enough to really advertise List-ID properly.

• Warn people about how much more mail they'll get when they subscribe to something structurally.

Out of scope

Some of the "Nice to have" elements above may move here.

• Subscribe to search results (related to 49752). This was originally part of the LEP, and advocated by mpt and jml. However, Deryck informed Gary that this was investigated and rejected for technical reasons. jml confirmed orally that he understood this and had accepted it.

• re-allow structural subscriptions on distributions. Summary: Gary thought this might be valuable. Deryck warned it had been a big problem in the past, both technical and social. Brian Murray said that he wanted this a lot; but then in later IRC discussion realized that he already had what he needed as long as we implement the filtering that we describe in the "Must" section. Excessive details follow.

  • Gary first wrote the following in the proposal.

    • Structural subscriptions to a distribution are currently disallowed except for a team (to allow mailing list subscriptions to all bug notices). Deryck reports that this was done for two reasons. The most important one was performance: sending out bugs for the structural subscriptions on Ubuntu was causing timeouts. A secondary reason was social: people were somehow subscribing to all of the bugs in Ubuntu, receiving tons of mail, and publicly complaining about the receipt of all those notifications (Launchpad was "spamming" them). We'd have to figure out a way to reduce the risk of a performance problem, but we could re-allow structural subscriptions on distributions. For instance, I could imagine people wanting to receive all bugs that match a certain tag. Perhaps we could allow subscriptions as long as they were constrined to certain tags?

  • Deryck replied in mail:

    • Allowing people to subscribe to all Ubuntu bugs creates lots of problems.

    • I realize people want to fix technical issues and leave social issues to communities, but in my time on bugs, this was always a source of problems. I fear that if you re-enable this in order to allow filtering on the distro at the highest level, you will end up creating even more unique opportunities for people to get themselves subscribed to a ton of bug mail and not be able to work out what happened.
    • I would certainly consider this out of scope since it enlarges the bug mail options, rather than reducing noise or providing more control. If this story is a great success, then leave it to someone else to work out how to add this back in. Also, I'm not sure anyone is really asking for this. It's only nice, IMHO, in a theoretical sense or for technical purity (i.e. you can subscribe to other pillars, so why not distros?).
  • Gary likes taking things entirely out of scope, and so he did so, with the understanding that the change was still part of the proposal for Jono to re-approve. However, Brian Murray then sent the following to the list, so Gary brought it back to nice-to-have.
    • Being able to structurally subscribe to Ubuntu bugs is exactly what I, and likely the Ubuntu QA team, want to do. I want to know about all the bug tasks that have been set to High or Critical for any Ubuntu package so that I can then evaluate them and if necessary bring them to the attention of the release team for Ubuntu by targeting them to a series / release. I currently do this by parsing the ubuntu-bugs mailing list which gets email regarding all public bug reports.
    • I can also imagine lots of ways people would want to be subscribed to bug reports about any package with a particular tag.
  • From Brian's reply, in fact, it sounded like it would be one of the first candidates to move from "Nice to have" to "Must". Deryck and Gary and Brian discussed this all briefly on IRC. It started with Deryck pointing out that Brian was one of the people who requested and coded the change to disallow subscriptions to distributions. This conversation followed:
    • gary_poster: so, bdmurray, you share deryck's social concern, but would really like the functionality for yourself?
    • gary_poster: IOW, as a clunky interface example, you would like to say that users *can* subscribe to distributions, but they get warned about it? Or...you have to be a member of some team?
    • bdmurray: gary_poster: yes, that's a fair statement my concern is people doing something they didn't intend and getting hate mail about it.
    • bdmurray: It'd be neat if you could see how many bugs would have met your criteria over the past 24 hours or a week
    • bdmurray: You could even just allow distro subscriptions via the API and I'd be happy
  • Later, bdmurray realized he already had what he needed himself. On the basis of this, Gary thinks this is still arguably a "nice to have" but he would prefer it to be "out of scope."
    • bdmurray: gary_poster: looking again I think bug supervisors can structurally subscribe to distro bugs if a bug supervisor is set
    • bdmurray: gary_poster: at least that is the way it was when I wrote it
    • gary_poster: bdmurray, oh? I'm afraid I'm not even sure how bug supervisors fit into it
    • bdmurray: gary_poster: its only for distributions with bug supervisors set that structural subscriptions are restricted for and they are restricted to the members of the bug supervisor team
    • bdmurray: gary_poster: so there is no need to "re-allow" structural subscriptions on distributions
    • gary_poster: ah! so you can already subscribe to distributions, so if we add filtering for that then that will allow you to use that again?
    • bdmurray: not again but yes
    • gary_poster: I thought you were not using that now, because it was too much?
    • bdmurray: Okay, yes *I* personally have not been subscribed to all ubuntu bugs for a long time now
    • gary_poster: I think I got you, then. thanks for clarifying

    • bdmurray: gary_poster: here is the mp https://code.launchpad.net/~brian-murray/launchpad/bug-556489-distro-struct-sub/+merge/31090

• There is a category of bug mail to do with duplicates that people might like to filter: for commonly-duplicated bugs I might not want to know about every one of the dupes coming in, or every comment on each of them.

Subfeatures

(none)

Workflows

Please see the more detailed workflows, including mockups, described in yellow/Subscriptions. The following are the workflows from the original LEP.

Bug page

New subscription:

• A user clicks "Subscribe"
• An overlay appears with the various options
• The user makes choices, clicks "subscribe" on the overlay
• The user is subscribed to the bug with those options

Existing subscriptions:

• A user clicks "Edit Subscription"
• An overlay appears with the various options
• The user changes the choices and clicks "Update subscription"
• The changes take effect

Packages, Projects

New subscription:

• A user clicks "Subscribe"
• An new page is loaded with the various options
• The user makes choices, clicks "subscribe" on the form
• The user is subscribed to the bug with those options and redirected back to the bugs homepage for the object

Existing subscriptions:

• A user clicks "Edit Subscription"
• An new page is loaded with the various options
• The user changes the choices and clicks "Update subscription"
• The changes take effect and the user is redirected to the bugs homepage for the object

Distributions

As mentioned above in the "Nice to have" section, structural subscriptions to a distribution are disallowed except for a team (to allow mailing list subscriptions to all bug notices). We might consider re-opening this, as discussed above.

Mockups

The current mockups can be seen in yellow/Subscriptions.

These are the original mockups: lp:~gmb/+junk/subscription-widget-mockups/

Success

How will we know when we are done?

All critical and high bugs for the story-better-bug-notification tag are closed. These bugs should either be related to the features in the Must or Must Not section, or technical bugs about the implementation of these features.

Features in the "Nice to Have" section may be promoted to "Must" by the Product Strategist at any time, but we are hoping that these will only be promoted when we have delivered, or are about to deliver, the full set of "Must" features. Then the product strategist with the yellow squad can determine if any further increments are desired before the LEP is closed and the squad moves to a bug rotation.

How will we measure how well we have done?

• We see an increase of bug subscriptions after the features are released, because users now want to use bug subscriptions more because they are more usable.
• We see significant use of the new features--the filters in particular.

Any other ideas? Other possibilities from the original LEP, which are currently not intended to be used:

• less bug mail for pitti, seb128
• less mail to ubuntu-bugs
• how many packages lack subscribers

  • On staging, 6818 Ubuntu source packages out of 34423 have subscribers [2010-04-21]

    • SELECT COUNT(DISTINCT sourcepackagename) FROM structuralsubscription WHERE distribution = 1;

    • SELECT COUNT(*) FROM sourcepackagename;

  • See also http://bazaar.launchpad.net/~ubuntu-bugcontrol/ubuntu-qa-tools/master/annotate/head:/launchpadlib-scripts/packages-without-subscribers.py

Useful info to have:

• what filters/options are selected?

Thoughts?

Unlimited configuration for notifications?

The following discussion was in an earlier version of the draft. It apparently happened before the team decided that subscription-from-search was out of scope. It seems that the first speaker is Deryck.

Dividing filtering

We decided against the following line of thinking, because it led to dividing "filters" off away from "subscriptions". We felt providing a unified interface to what was called was both more understandable and more flexible. In our revised design, you can subscribe multiple times to the notifications on a particular bug target. On each subscription, you can filter based on both the kind of events (which was available on "subscriptions") and on the state of the bug (importances, statuses, and tags, which were available on "filters").