launchpad development|
Size: 10482
Comment: Wording tweak in summary bar.
|
Size: 11325
Comment: refresh per list discussion
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 2: | Line 2: |
| = A Practical Guide to Launchpad Bug Triage = ''Or: "Why I don't classify bugs as medium"'' (Based on an email from Curtis Hovey.) The process of triaging issues (bugs, features, and tasks) has one crucial principle: Prioritise the work according to ''need'' and ''certainty''. Work is prioritised because there are not enough engineers to do all the work. Some features will never be completed, some bugs will never be fixed. Triage determines which bugs can and will be fixed, which features can and will be implemented. ''Need'' is generally understood, when planning work, but ''certainty'' is not, and that often leads to wasted work and unmet expectations. By ''need'', I mean a measure of severity. What percentage of users does the issue affect, and how severely does it impede them from completing their task. By ''certainty'', I mean a measure of how certain the engineers are that they can address the issue. Time is also a factor in this measure, the longer an issue takes to address, the more likely that the conditions that were first judged will change. The act of triage is separating work into groups that are being worked on now, next and last. There can only be as many "now" bugs or features as there are engineers. The number of "next" work is limited to the velocity of the engineers and how infrequently plans change. The bugs that are last will probably never be addressed, the last features may never be started. The corollary to this rule is that there are a finite number of bugs or features in the first two groups. There cannot be more work in these groups than there are engineers to do for the given period of time; otherwise the engineers, businesses and users are being misinformed about when issues will be addressed. == An Example == Consider there is one engineer and two bugs. He can only work one bug at a time. One bug is more important than the other. The risk is that he may not be able to fix one of the bugs before users are disappointed and abandon the application. He risks disappointing all users if he does not fix either bug because he choose the one with the most need over the one he was certain he could address. If he does not know how to fix the bug with the most need, or that the fix takes a long time, he is wasting time he could have spent fixing the bug with more certainty. The only way he can address the bug with the most need is to employ a hack to reduce the need, to meet the expectations of some users. The hack is also used to gain time to understand the problem, thus increase certainty. == Only Assign Work that You Are Commiting to do in the Near Future == When a work is assigned to an engineer, he is commiting to complete the work in the near future. What the "near future" means is different for each project. I suggest 3 releases is the "near future", because when work is planned, the engineer is thinking about now, next, and last. For some projects this period might be 6 weeks, for others, 6 months. I prefer to plan for the current release, and the next one. As work is reprioritised, it may be rescheduled to the third release. I do not think it is wise to plan a bug or feature to be completed in the third releases because if it slips to the fourth or fifth released, I doubt the it was correctly prioritized as high. Any high work that is assigned to a engineer for more than 3 releases was not high. If it were, the work would have been reassigned to someone who could complete it in the scheduled time. Any other work that is assigned for more than 1 release is also misprioritised. You are lying to yourself, and the the project's users, when you assign work that you are not committing to fixing. == Practical Classifications of Importance == Work is often classified in relative terms. It is better to classify work according to how it are managed to convey when and under what terms the bug will be fixed or a feature will be complete. There are three priorities that work can be classified as: Critical:: :: The bug dramatically impairs users. Users may lose their data. Users cannot complete crucial tasks. The feature is needed to encourage adoption or prevent abandonment of the project. <<BR>> :: Synonyms: required, essential, now, must do <<BR>> :: The work is immediately assigned to a engineer. It is his top priority to fix. Team members help the engineer to plan and do the work. The work is released as soon as it is deployable; in the case of a bug, it is released outside of the release schedule. High:: :: The bug prevents users from completing their tasks. The feature provides new kinds of tasks or new ways of completing tasks. <<BR>> :: Synonyms: expected, next, can do, should do <<BR>> :: The work is assigned to a engineer to be completed in the next 3 releases. The engineer may choose to do other work if he believes it is within the scope of the high priority work. Medium:: :: The bug is inconvenience for many users. The feature provides new ways of completing tasks. <<BR>> :: Synonyms: preferred <<BR>> :: The work is not scheduled, though it is intended to be completed. When the work is assigned, it may also be scheduled, but there is no commitment to complete it for the stated release. The engineer may choose to postpone the work in favour of more important work. Low:: :: The bug is an inconvenience to users, but it does not prevent them from completing their tasks. The feature is a convenience to users. <<BR>> :: Synonyms: optional, last, may do <<BR>> :: The engineer may assign the work to himself while working on a high priority work because the high work provides an opportunity to complete the low priority work at less cost. If the low work in any way jeopardises the high priority work, the low work is unassigned. The engineer is thus ''certain'' that the work can be fixed quickly and without difficulty. A corollary to this rule is that low work that is assigned to a engineer must be "in progress" or "fixed" states. == The Problem with "Medium" == It might be argued that when the engineer has an opportunity to fix a low or a medium bug, he must choose the medium one. This rules does not define a practical distinction between medium and low. There is no commitment to fix the medium bug; it will not be scheduled for fixing. A engineer chooses to undertake a low bug because he sees an opportunity to fix it while working in the affected code. The engineer is choosing to do unscheduled work because he is ''certain'' it does not jeopardise his scheduled work. The engineer might see an opportunity to fix a medium and a low bug at the same time, but that is unlikely. It can also be argued that 'critical' is 'high' and that 'high' is 'medium'. True, that is a matter of semantics. The crux of the issue is that there are three practical classifications of work. The words chosen to describe the classifications could use the tofu scale of hard, firm, and soft. People who are unfamiliar with triage will appreciate names that convey the kind of attention the issue will receive. Some teams with a large number of bugs prefer to keep a pool of medium work from which releases are planned. Items in the pool may be escalated to high if it is perceived that once work is started, there should be a commitment to complete it as scheduled. This work is different from low work because the work makes a substantial improvement to the application, but like low, there is no commitment when the work will be completed. It can be argued that work starts on medium bugs and features because of changes to other priorities, certainties, or the number of users it affects. == Consequences of Misprioritised Work == Stakeholders often use reports that list the prioritised work for a release and for each engineer. When work is misclassified there are two commonly observed consequences: a decreased in certainty, and a decrease in communication. In the first consequence, the engineer's effort may be wasted; there are issues that have more ''need and certainty''. Engineers, and other stakeholders, are often tempted to complete the misdirected work after the misclassification is discovered because it is assumed that it is better to always deliver something finished than nothing at all. This is a risky choice, because it jeopardises work in future releases. By working on less important work, the engineer is decreasing the certainty of the more important work. The second consequence is that the engineer ignores the list and he works on issues according to some other source, such as the opinion of another stakeholder. While the engineer is working on the correct issue, it is unclear to other parties what work is going on and when will it be completed. Users may abandon the project in frustration. Planners cannot coordinate all the stakeholders. The first consequence is possibly a failure to do re-prioritisation during the triage process, but second consequence is a total failure in the triage process. Why would anyone do triage if the prioritisation will be ignored? How can work be coordinated if the work is unknown to all stakeholders? Why would users trust a project if it does not do what it says it will do? Work must be reprioritised during the triage process to ensure that engineers are working on the issues with the most need and certainty. Engineers must work from the list or prioritised issues. == Indicators of Misprioritised Work == The rules of practical classification provide tests for misprioritised bugs, features, or tasks. * The work is critical, but it is not assigned and targeted for release. * The work prioritised as high, but it is not assigned and for a release. * The work is high, but have not been worked on in 3 releases. * The work is low and unassigned, yet it is targeted for a release. * The work is low and assigned, but the engineer is not working on it. * The work is considered to be triaged, but it's priority is not critical, high, or low. * An engineer is assigned more work than he can accomplish in 3 releases, and it cannot be reassigned. |
= Launchpad Bug Triage = == What is bug triage? == Triage is the act of sorting bugs into different priority groups. There are many conflicting sorts - everyone has their pet bug that should be 'first'. The sort order we choose is from the projects perspective: we try to balance the needs of our users. So, bug triage is: '''sorting bugs by importance-to-the-project''', and these are the influences we try to strike a balance between in assessing that importance: * Things affecting launchpad project health. * Things affecting stakeholders * Things affecting other users When we have triaged a bug, it has status '''triaged''' and an importance other than '''unknown'''. == Why triage == This may be obvious, but having just a big bucket of open bugs isn't very efficient: there are more genuinely important issues to fix than engineers, and as such engineers will forget what things are urgent and what aren't. Secondly, each of the groups of users whose needs we're trying to compromise between are interested in when things will get done. By sorting the bugs we provide a proxy metric for when tasks will be worked on. == How much triage is needed? == The world is dynamic and constantly changing; as such any sort we come up with for our bugs will be outdated pretty quickly. We could make the sort complete (so all bugs are ranked) and constantly refresh it. However this is inefficient: the only times the sort actually matters are: * when a new bug is being selected to work on (by project importance). * when a user is taking a decision based on how long until the bug is likely to be worked on. For instance, they might decide to work up a patch, or whether to use Launchpad at all. So how much sorting is enough? Two interesting metrics are freshness and completeness. If the sort is too old, bugs will be indicated as 'should be next to work on' that are not valid as that any more. Our priorities may change month to month but they rarely change faster than that : so we can tolerate things being months (or more) stale. The sort is complete enough if the answers to 'what is an important bug to work on now' and questions that users may ask (like 'how long till this will be worked on') get answers accurate enough... and how accurate do we need? Well thats a tradeoff, but we think the answers are accurate enough if: - users can see that we care about performance, regressions, usability and polish - engineers selecting 'next bug to work on' based on the triage sort usually pick things that are the most useful thing to the project/stakeholders/users; that is that inconsequential stuff is tackled after consequential stuff == Bug Importance == Bug importance in Launchpad is where we record the result of the triage process; we have 5 buckets we can use in Launchpad: critical/high/medium/low/wishlist. We don't actually ever block a release based on having a particular importance bug - we block releases based on having regressions, which any commit can have - and we mark that on the bug mapping to the commit. The buckets combine to give a partial sort: bugs in the critical bucket are sorted before bugs in the high bucket. We can choose to use some or all of these 5 buckets. How many do we need? A good way to answer that is to consider our hypothetical complete, fresh sort, and consider how many slices we'd need to make in it to answer questions well; we also need to consider what would change to those slices when things change (such as new things coming that sort to the front). Also buckets have a cost : we need a ruleset for triage that will let us assign bugs to buckets: every bucket makes the heuristics more complex. Given that we have a freshness tolerance for most bugs of some months, that we don't want to update many bugs when a single bugshuffles in front, and that because we have more bugs coming in than we fix - we need three or perhaps four buckets: * A topmost bucket that is generally empty and crisis bugs go into. * A default bucket that bugs we haven't picked out as being important enough to sort above any other specific bug go into. * [optional] a bucket for bugs that are reasonably important but not extremely so * And a bucket containing bugs which are within the first 6 months of work We map these buckets into: * critical : generally empty, bugs that need to jump the queue go here. * high: bugs that are likely to get attention within 6 months * low [or perhaps wishlist]: All other bugs. This has a clear tension: time-till-we-start-work is a good metric for what bucket to put in, but given a bug with some symptoms how do we decide what bucket it should go into. To address this tension we use two things: * A quarterly review of the bugs in the high bucket, to stop it overflowing. * Some heuristics for sorting bugs == Quarterly review == This is pretty simple - we re-triage bugs with high importance to see if things have changed and they should be downgraded. For upgrades we assume that user prompting will cause us to upgrade them. == Triage guidelines == These guidelines describe the rules we use to sort bugs - and from that sort we assign bugs to bugs. We broadly want: * queue jumping bugs to be in the critical bucket. (OOPS, timeouts, regressions, stakeholder-escalated bugs are all examples of queue jumping bugs) * the high bucket to be about 6 months deep - many parts of Canonical are on a 6-month cycle and fitting in with that is convenient The quarterly review is responsible for shrinking the high bucket if its too full. What we need to do then in assessing the bucket for a bug is to do *enough* sorting on it to see if its a queue jumper, of its its more important than the least important bug currently in the high bucket. Beyond that, all bugs are in the low bucket. If a bug is a regression : if the thing *was* working and now isn't, we sort it higher. We're currently discussing having a policy that regressions are critical, which if implemented will make these queue jumpers (critical bucket). If the bug is one that has been escalated via the Launchpad stakeholder process, it is a queue jumper (critical bucket). OOPS and timeout bugs also jump the queue: performance is very important to our stakeholders and OOPS dramatically affect our ability to operate and maintain Launchpad as well as being a very negative experience when encountered. The ZeroOopsPolicy contains details on this. For things like browser support, when a new browser is released but the vendor is in our supported-browser-set, we should treat issues as regressions and so they will be queue jumpers. Beyond these rules a bug is more important than another bug if fixing it will make Launchpad more better than fixing the other bug. Discretion and a feel for whats in the bug database will help a lot here, as will awareness of our userbase and their needs. One sensible heuristic is to look at 5-10 existing high bugs, and if the new bug is less important than all of them, mark it low (its probably less important than all existing high bugs). Engineers have discretion to decide any particular bug should be sorted higher (or lower) than it has been; some change requests are very important to many of our users while still not big enough to need a dedicated feature-squad working on them (so these bugs may be high). When two engineers disagree, or if someone in the management chain disagrees, common sense and courtesy should be used in resolving the disagreement. == How to triage == Visit [[https://bugs.launchpad.net/launchpad-project/+bugs?field.importance=Unknown|unknown importance bugs]] and [[https://bugs.launchpad.net/launchpad-project/+bugs?field.status:list=NEW&field.status:list=INCOMPLETE_WITH_RESPONSE&field.status:list=INCOMPLETE_WITHOUT_RESPONSE|untriaged status bugs]] For each bug: * See if there are any duplicates by having a bit of a look around, search your memory etc. If you find a duplicate, mark the the '''newer''' bug as a duplicate of the '''older''' bug (unless there is a compelling reason to use the newer bug as the master. Consider updating the description and tags of the '''older''' bug to help make it clearer. We use the '''older''' bug by default because we (roughly) work through bugs in the same bucket in date order. * If the bug is unrelated to Launchpad, move it somewhere appropriate. * If the bug is something we won't do at all, mark it as won't fix. * If its a operational request, convert it to a question. * apply the guidelines in 'Triage Guidelines' to get a bucket for the bug and set the bug importance to that bucket. * If the bug status is 'Incomplete', check that the filer was asked to clarify something; if they were and haven't replied in a month, close the bug. Otherwise either ask them to clarify something, or set the bug to Triaged if they have clarified whatever was needed. * If the bug status is New, set it to triaged. == Assignment == Bug triage does not involve assigning an engineer. Engineers should only be assigned to bugs that are ''in progress''. Even critical bugs do not need an engineer assigned: operational incidents are not tracked in the bug database, though critical bugs may be generated as followup work to be done; those bugs are then in the front-section of the queue, but thats all that is needed. == Selecting bugs to work on == The bug database holds the /project/ importance set of bugs. However individual or squad work-queues may be quite different. For instance, we have 3 squads working on features at any one time, 2 on maintenance. Generally speaking squads on feature-rotation will ignore 'importance' in selecting what to work on - they will be working on a feature and creating bugs as appropriate to create discussion points and todo items for that feature. The Launchpad maintenance squads however will usually be working from the bug database - picking bugs up to work on based on their ''triaged importance''. So for maintenance squads, they should simply look in each bucket in order - critical, high, low - and from within that bucket take one of the oldest bugs - one that seems interesting to them at the time. Crucially though, all bugs in the critical bucket should have someone or some squad working on them before any bugs in the high bucket are picked up and worked on, and likewise for low. Community work will often ignore our bug triage and focus on itch scratching - and this also applies to patches done by Launchpad engineers in their personal and slack time: the selection logic for picking a bug only applies to effort being put in as part of their primary duties. That is, its always totally ok to fix that low priority bug thats really annoying you, whether you're a user of Launchpad or a developer. A bug fix is a bug fix! |
This page is about triaging Launchpad-related bugs. For general information on handling bug reports, see BugHandling. If you have any questions, ask for help right away. |
Launchpad Bug Triage
What is bug triage?
Triage is the act of sorting bugs into different priority groups. There are many conflicting sorts - everyone has their pet bug that should be 'first'. The sort order we choose is from the projects perspective: we try to balance the needs of our users.
So, bug triage is: sorting bugs by importance-to-the-project, and these are the influences we try to strike a balance between in assessing that importance:
- Things affecting launchpad project health.
- Things affecting stakeholders
- Things affecting other users
When we have triaged a bug, it has status triaged and an importance other than unknown.
Why triage
This may be obvious, but having just a big bucket of open bugs isn't very efficient: there are more genuinely important issues to fix than engineers, and as such engineers will forget what things are urgent and what aren't.
Secondly, each of the groups of users whose needs we're trying to compromise between are interested in when things will get done. By sorting the bugs we provide a proxy metric for when tasks will be worked on.
How much triage is needed?
The world is dynamic and constantly changing; as such any sort we come up with for our bugs will be outdated pretty quickly. We could make the sort complete (so all bugs are ranked) and constantly refresh it. However this is inefficient: the only times the sort actually matters are:
- when a new bug is being selected to work on (by project importance).
- when a user is taking a decision based on how long until the bug is
likely to be worked on. For instance, they might decide to work up a patch, or whether to use Launchpad at all.
So how much sorting is enough? Two interesting metrics are freshness and completeness.
If the sort is too old, bugs will be indicated as 'should be next to work on' that are not valid as that any more. Our priorities may change month to month but they rarely change faster than that : so we can tolerate things being months (or more) stale.
The sort is complete enough if the answers to 'what is an important bug to work on now' and questions that users may ask (like 'how long till this will be worked on') get answers accurate enough... and how accurate do we need?
Well thats a tradeoff, but we think the answers are accurate enough if:
- - users can see that we care about performance, regressions,
usability and polish
- - engineers selecting 'next bug to work on' based on the triage sort
usually pick things that are the most useful thing to the project/stakeholders/users; that is that inconsequential stuff is tackled after consequential stuff
Bug Importance
Bug importance in Launchpad is where we record the result of the triage process; we have 5 buckets we can use in Launchpad: critical/high/medium/low/wishlist.
We don't actually ever block a release based on having a particular importance bug - we block releases based on having regressions, which any commit can have - and we mark that on the bug mapping to the commit.
The buckets combine to give a partial sort: bugs in the critical bucket are sorted before bugs in the high bucket.
We can choose to use some or all of these 5 buckets.
How many do we need? A good way to answer that is to consider our hypothetical complete, fresh sort, and consider how many slices we'd need to make in it to answer questions well; we also need to consider what would change to those slices when things change (such as new things coming that sort to the front).
Also buckets have a cost : we need a ruleset for triage that will let us assign bugs to buckets: every bucket makes the heuristics more complex.
Given that we have a freshness tolerance for most bugs of some months, that we don't want to update many bugs when a single bugshuffles in front, and that because we have more bugs coming in than we fix - we need three or perhaps four buckets:
- A topmost bucket that is generally empty and crisis bugs go into.
- A default bucket that bugs we haven't picked out as being important
enough to sort above any other specific bug go into.
- [optional] a bucket for bugs that are reasonably important but not
extremely so
- And a bucket containing bugs which are within the first 6 months of work
We map these buckets into:
- critical : generally empty, bugs that need to jump the queue go here.
- high: bugs that are likely to get attention within 6 months
- low [or perhaps wishlist]: All other bugs.
This has a clear tension: time-till-we-start-work is a good metric for what bucket to put in, but given a bug with some symptoms how do we decide what bucket it should go into.
To address this tension we use two things:
- A quarterly review of the bugs in the high bucket, to stop it overflowing.
- Some heuristics for sorting bugs
Quarterly review
This is pretty simple - we re-triage bugs with high importance to see if things have changed and they should be downgraded. For upgrades we assume that user prompting will cause us to upgrade them.
Triage guidelines
These guidelines describe the rules we use to sort bugs - and from that sort we assign bugs to bugs. We broadly want:
- queue jumping bugs to be in the critical bucket. (OOPS, timeouts,
regressions, stakeholder-escalated bugs are all examples of queue jumping bugs)
- the high bucket to be about 6 months deep - many parts of Canonical
are on a 6-month cycle and fitting in with that is convenient
The quarterly review is responsible for shrinking the high bucket if its too full.
What we need to do then in assessing the bucket for a bug is to do *enough* sorting on it to see if its a queue jumper, of its its more important than the least important bug currently in the high bucket. Beyond that, all bugs are in the low bucket.
If a bug is a regression : if the thing *was* working and now isn't, we sort it higher. We're currently discussing having a policy that regressions are critical, which if implemented will make these queue jumpers (critical bucket).
If the bug is one that has been escalated via the Launchpad stakeholder process, it is a queue jumper (critical bucket).
OOPS and timeout bugs also jump the queue: performance is very important to our stakeholders and OOPS dramatically affect our ability to operate and maintain Launchpad as well as being a very negative experience when encountered. The ZeroOopsPolicy contains details on this.
For things like browser support, when a new browser is released but the vendor is in our supported-browser-set, we should treat issues as regressions and so they will be queue jumpers.
Beyond these rules a bug is more important than another bug if fixing it will make Launchpad more better than fixing the other bug. Discretion and a feel for whats in the bug database will help a lot here, as will awareness of our userbase and their needs. One sensible heuristic is to look at 5-10 existing high bugs, and if the new bug is less important than all of them, mark it low (its probably less important than all existing high bugs).
Engineers have discretion to decide any particular bug should be sorted higher (or lower) than it has been; some change requests are very important to many of our users while still not big enough to need a dedicated feature-squad working on them (so these bugs may be high). When two engineers disagree, or if someone in the management chain disagrees, common sense and courtesy should be used in resolving the disagreement.
How to triage
Visit unknown importance bugs and untriaged status bugs
For each bug:
- See if there are any duplicates by having a bit of a look around,
search your memory etc. If you find a duplicate, mark the the newer bug as a duplicate of the older bug (unless there is a compelling reason to use the newer bug as the master. Consider updating the description and tags of the older bug to help make it clearer. We use the older bug by default because we (roughly) work through bugs in the same bucket in date order.
- If the bug is unrelated to Launchpad, move it somewhere appropriate.
- If the bug is something we won't do at all, mark it as won't fix.
- If its a operational request, convert it to a question.
- apply the guidelines in 'Triage Guidelines' to get a bucket for the
bug and set the bug importance to that bucket.
- If the bug status is 'Incomplete', check that the filer was asked
to clarify something; if they were and haven't replied in a month, close the bug. Otherwise either ask them to clarify something, or set the bug to Triaged if they have clarified whatever was needed.
- If the bug status is New, set it to triaged.
Assignment
Bug triage does not involve assigning an engineer. Engineers should only be assigned to bugs that are in progress. Even critical bugs do not need an engineer assigned: operational incidents are not tracked in the bug database, though critical bugs may be generated as followup work to be done; those bugs are then in the front-section of the queue, but thats all that is needed.
Selecting bugs to work on
The bug database holds the /project/ importance set of bugs. However individual or squad work-queues may be quite different. For instance, we have 3 squads working on features at any one time, 2 on maintenance. Generally speaking squads on feature-rotation will ignore 'importance' in selecting what to work on - they will be working on a feature and creating bugs as appropriate to create discussion points and todo items for that feature.
The Launchpad maintenance squads however will usually be working from the bug database - picking bugs up to work on based on their triaged importance. So for maintenance squads, they should simply look in each bucket in order - critical, high, low - and from within that bucket take one of the oldest bugs - one that seems interesting to them at the time. Crucially though, all bugs in the critical bucket should have someone or some squad working on them before any bugs in the high bucket are picked up and worked on, and likewise for low.
Community work will often ignore our bug triage and focus on itch scratching - and this also applies to patches done by Launchpad engineers in their personal and slack time: the selection logic for picking a bug only applies to effort being put in as part of their primary duties. That is, its always totally ok to fix that low priority bug thats really annoying you, whether you're a user of Launchpad or a developer. A bug fix is a bug fix!
