* '''Launchpad entry:''' https://launchpad.net/products/malone/+spec/search
 * '''Created:''' <<Date(2005-12-12T20:01:31Z)>> by MatthewPaulThomas 
 * '''Contributors:''' MatthewPaulThomas

This is a functional specification, not an incremental one. It describes many small features (most of which can be implemented independently), so we can tell whether the features are logically consistent and work together to satisfy likely use cases.

<<TableOfContents>>

== Summary ==

To cope with vast numbers of bug reports, Launchpad’s bug searching abilities should include functions available in modern search engines and other bug trackers: defaulting to case-insensitive substrings, searching all open bug reports, boolean and other operators for text-only advanced searches, phrase searches and bug number disambiguation, advanced operator suggestions, spelling and punctuation canonicalization, ignoring stop words by default, and spelling suggestions. The text syntax for advanced searches should make results pages simpler, by letting the search be displayed as a line of text. This would subtly train people in the advanced search syntax, but Launchpad should also link to a page of search tips.

== Rationale ==

The most common function in any bug tracker is searching — finding bugs that need work, and checking whether bugs have already been reported. This is even more important with Launchpad, which will eventually contain bug reports about thousands of products, tens of thousands of packages, and dozens of distributions, and which will be used by hundreds of thousands of people of varying abilities. So Launchpad's search must be very powerful and very smart.

== Use cases ==

 * Matt is looking for a bug he reported earlier about Network``Manager Being in too much of a hurry to use the advanced search form, or even to navigate to Malone, he enters `network manager reporter:mdz` in the search field on Launchpad's front page. Launchpad returns the bug report Matt was looking for, despite it mentioning “Network``Manager” but not “manager” alone.

 * While translating Word``Press, Pieter-Jahn notices that one of the strings is “Click here to login!”. This really should be "Log in here". So Pieter-Jahn searches Word``Press bug reports for `"click here to login"`, and Malone returns only bug reports containing that phrase.

 * Diogo is looking for bug reports about 404 errors in Launchpad itself. At the Launchpad project’s Bugs page he types `404`. Launchpad takes him directly to [[https://launchpad.net/malone/bugs/404|bug #404]], which is not what he meant. Fortunately, the top of the page features a section saying ''Were you looking for: __bug reports containing "'''404'''"__?'' From seeing `"404"` in the search field on the results page, he learns that the quick way of searching for a number is to put the number in quotes.

 * Sebastien is chief Gnome packager for a famous Linux-based OS. He wants to see only bugs that are OS-specific (aren’t recorded as occurring upstream), or that are filed upstream but have been fixed upstream or in another OS. To do this, he goes to his OS’s Bugs page, searches for `-project: | elsewhere:fixed`, and bookmarks the result.

 * Ernie wants Mutter to include, by default, a fast window-switching mechanism like Mac OS X has. Being unfamiliar with Launchpad's context-sensitive nature, he does this by typing `mutter expose` at Launchpad's main Bugs page. Fortunately Launchpad is wise to this, and at the top and bottom of the search results, presents Ernie with a link: ''Are you looking for: __bugs in '''Mutter''' only__?'' Following that link, Ernie is delighted that Launchpad returns a bug mentioning "Exposé" even though he didn't type the accent.

 * Marina finds that `gnome-panel` doesn't display properly after being resized, but she doesn't know it's called `gnome-panel`. At Launchpad's main Bugs page she types `the panel doesnt display after I resize it`. Launchpad is smart enough to include results for ''doesn't'' as well as ''doesnt''. It is also smart enough ''not'' to ask ''Are you looking for: __bugs in '''THE''' only__?'', because The Hessling Editor is on its blacklist of packages with very common words as names.

 Launchpad does, however, tell Marina that it is showing results for ''panel doesnt display after resize'', having ignored the words "the", "I", and "it" because they are unimportant and very common in bug reports and the search string included at least one uncommon word. `the panel doesnt display after I resize it` is still displayed in the text field at the ends of the results page, in case Marina wants to put a `+` in front of one or more of the common words, or search for them as a phrase.

 * Tarina, who was separated from Marina at birth, searches for `the panel doesnt displqy after I resize it`. Launchpad, after checking its list of words used in bug reports, and checking the usefulness of its suggestion ahead of time, suggests: ''Did you mean: __the panel doesn't '''display''' after I resize it__?''

== Implementation progress ==

{o} unimplemented<<BR>>
/!\ in progress<<BR>>
{*} implemented

== Defaulting to case-insensitive substrings ==

 * [[https://launchpad.net/products/malone/+bug/2795|#2795: Bug searching should default to case-insensitive substrings]]

{o} Like Bugzilla has always done, Launchpad should default to searching for case-insensitive substrings -- so that, for example, searching for `mail` finds bug reports containing ''e-mail'', ''email'', ''$MAILER'', ''mailing'', or ''mailed''. {o} Exact words can be found using the phrase search; for example, searching for `"mail"` finds only bug reports containing the exact word ''mail''. (This helps satisfy Matt's use case.)

== Searchable text ==

 * [[https://launchpad.net/bugs/1265|#1265: Bug search seems to ignore matching on assignee]]
 * [[https://launchpad.net/bugs/1440|#1440: Distribution bug search doesn't include package name]]
 * [[https://launchpad.net/bugs/5782|#5782: Searching for "upstream" doesn't do anything]]

{o} The searchable text for a bug report should be its summary, description, context names, and context display names. It should not include reporters, assignees, statuses, importances, or milestones; these are handled by operator suggestions. Nor should it include comments; searchworthy comment text should be included in updated bug descriptions. (This helps satisfy Ernie's use case.)

== Searching all open bugs ==

 * [[https://launchpad.net/bugs/30369|#30369: malone front page should have search option]]

{o} `https://bugs.launchpad.net/` (and, before that host exists, `https://launchpad.net/bugs`) should feature a form letting people search for bug reports in any product, package, or distribution across Launchpad. (This helps satisfy Matt's, Ernie's, Marina's, and Tarina's use cases.)

== Boolean operators ==

 * [[https://launchpad.net/bugs/6059|#6059: Option to ignore merge new package "bugs"]]

{o} By default, Launchpad should search for bug reports containing all the terms searched for (except for stop words). {o} Searches for any of a group of terms should be enterable using `|` or `OR`. Without brackets, this applies only to the two words and/or phrases on either side of the operator: for example, ''print dialog OR window'' should return bug reports containing ''print'' and either ''dialog'' or ''window''. This should be alterable using brackets: ''(print dialog) OR window'' should return bug reports containing ''print'' and ''dialog'', and also bug reports containing ''window''.

{o} Unmatched brackets should be ignored for the purpose of the search. They should still be presented in the input field for editing, but should be omitted from the search string displayed next to the number of results.

{o} A `-` sign in front of any search term should exclude bugs matching that term. For example, `-wrap` should exclude bug reports containing the word ''wrap'', and `dialog -(save OR print)` should return bug reports about dialogs other than the Save and Print dialogs. {o} Boolean operators should also be usable with advanced operators and phrase searches, once they are implemented; for example `-reporter:mdz` should exclude bugs reported by Matt Zimmermann, and `-"merge new package"` should exclude bug reports asking for new packages to be merged.

== Advanced operators ==

 * [[https://launchpad.net/products/malone/+bug/1006|#1006: Cannot determine how to search for submitter]]
 * [[https://launchpad.net/products/malone/+bug/5594|#5594: Malone should support advanced search syntax, e.g. "assignee:bradb"]]
 * [[https://launchpad.net/products/malone/+bug/6572|#6572: In distribution bug searches, it should be possible to filter out bugs with upstream tasks]]
 * [[https://launchpad.net/bugs/28698|#28698: "Fix committed" bugs are treated as "resolved" bugs]]
 * [[https://launchpad.net/bugs/74067|#74067: Limit bug searches by package repository]]

It should be possible to enter advanced operators anywhere in a search string. (This helps satisfy Matt's use case.)

||<style="width: 8em;"> '''Operator'''||'''Examples'''||
|| {o} assignee:||`assignee:` `assignee:mpt` `assignee:mpt@``myrealbox.com` `assignee:mpt@``canonical.com`||
|| {o} attachment:||`attachment:` `attachment:patch` `attachment:image/*`||
|| {o} cc:||`cc:mpt` `cc:mpt@``myrealbox.com` `cc:mpt@``canonical.com`||
|| {o} cve:||`cve:` `-cve:` `cve:1999-0067`||
|| {o} distro:||`distro:` `distro:impilinux` `distro:ubuntu/dapper` `distro:ubuntu/5.10`||
|| {o} duplicate:||`duplicate:` `duplicate:1234`||
|| {o} elsewhere:||`elsewhere:` `elsewhere:invalid,wontfix` `elsewhere:open` `elsewhere:fixed`||
|| {o} importance:||`importance:` `importance:high` `importance:>medium` `importance:<=low`||
|| {o} involving:||`involving:mpt` `involving:mpt@``myrealbox.com` `involving:mpt@``canonical.com`||
|| {o} keyword:||`keyword:design` `-keyword:privacy`||
|| {o} project:||`project:` `project:firefox` `project:firefox/2.0`||
|| {o} projectgroup:||`projectgroup:gnome` `-projectgroup:mozilla`||
|| {o} package:||`package:` `package:gnome-panel` `package:ubuntu/mozilla-firefox`||
|| {o} private:||`private:`, `-private:`||
|| {o} reporter:||`reporter:mpt` `reporter:mpt@``myrealbox.com` `reporter:mpt@``canonical.com`||
|| {o} repository:||`repository:main` `-repository:restricted`||
|| {o} status:||`status:fix-released` `status:open` `-status:fixed` `status:any`||
|| {o} tag:||`tag:` `tag:crash,crasher` `-tag:silly`||

The operators `attachment:`, `cve:` `distro:`, `importance:`, `package:`, `project:` and `tag:` should be usable standalone, and the operator `private:` should work only when used standalone. Where an operator is used standalone, it should filter based on whether the relevant property exists at all. For example, `assignee:` should return only those bugs that are assigned to someone, `cve:` should return only those bugs that are associated with a CVE, and `product:` should return only those bugs that are recorded as occurring upstream. (`-project:` satisfies Sebastien's first use case, by returning bugs that are not associated with a project.)

{o} For any advanced operator that takes values, either "`,`" or "`|`" should work to enter multiple values. For example, `status:unconfirmed,needs-info` should return bug reports with statuses of either Unconfirmed or Needs Info. `status:open`, a shortcut value for `status:unconfirmed,confirmed,needs-info,in-progress`, should be an implicit default (except on a person's Bugs pages), so the shortcut value `status:any` should be usable to include resolved bugs as well. There should be four shortcut values in total:
||'''Shortcut value'''|| {o} `open`|| {o} `any`|| {o} `development`|| {o} `fixed`||
||'''Unconfirmed'''   ||     *     ||     *    ||                  ||            ||
||'''Needs Info'''    ||     *     ||     *    ||                  ||            ||
||'''Confirmed'''     ||     *     ||     *    ||         *        ||            ||
||'''In Progress'''   ||     *     ||     *    ||         *        ||            ||
||'''Fix Committed''' ||           ||     *    ||         *        ||      *     ||
||'''Fix Released'''  ||           ||     *    ||                  ||      *     ||
(`elsewhere:fixed` satisfies Sebastien's second use case, by returning bugs that are not fixed in his distro but are fixed somewhere else.)

{o} If a search string contains a substring of the form `x:y` where ''x'' is not recognized, `x:y` should be passed through as a phrase included in the search string (presented as ''"x y"'' next to the results count). {o} If ''x'' is a known operator but ''y'' is an unknown value, it should similarly be treated as a phrase, but with an added warning ''y is not a recognized x'' (or, for the person operators, ''y is not a registered person''). {o} Either way, unknown operators should also be subject to spell-checking suggestions.

{o} Switching from the advanced to the simple search form should convert the graphical operators to the above text syntax. Conversion the other way should not be done, because it would be unexpectedly inconsistent (not happening for operators embedded in boolean expressions, for example, and not maintaining search term order on a round-trip conversion).

== Phrase searches ==

 * [[https://launchpad.net/bugs/56244|#56244: Can't search for phrases in bug reports]]

{o} Putting a phrase in `""` quotes should cause Launchpad to search for the phrase rather than the individual words. (This satisfies Pieter-Jahn's use case.) {o} As with unmatched brackets, unmatched quotes should be ignored.

== Bug number disambiguation ==

 * [[https://launchpad.net/products/malone/+bug/5943|#5943: can't search for numbers in Malone]]

{o} Similarly, putting a number in in `""` quotes should cause Launchpad to search for the number rather than going to that bug report. {o} For people who don't realize this, when someone enters a number and Launchpad goes directly to the bug report, the question ''Were you looking for: __bug reports containing "'''number'''"__?'' should appear at the top of the bug report. (This satisfies Diogo's use case.)

== Advanced operator suggestions ==

 * [[http://launchpad.net/bugs/5447|Bug 5447: Entering a product name at a package bugs page doesn't offer the product]]
 * [[http://launchpad.net/bugs/174394|Bug 174394: Distribution Bugs search should suggest package-specific searches]]

{o} No matter what context the search form is in, if the name of a distribution, distribution release, project, product, package, or bug status, or the ID or e-mail address of a person is present in a search string, Launchpad should present up to two suggestions to refine the search. For example:

 ||'''No''' bug reports matching '''ubuntu iwj firefox pdf plugin'''.||

 ''Are you looking for: __bugs in '''Ubuntu'''__? or __bugs assigned to '''Ian Jackson'''__?''

{o} These links should be residual: if you follow the ''__bugs in '''Ubuntu'''__'' link in that example, links for ''__bugs assigned to '''Ian Jackson'''__'' and ''__bugs about '''Firefox'''__'' are available on the next results page, so you can progressively narrow down the search.

{o} Launchpad should not make suggestions based on things named after stop words. Launchpad admins should be able to edit the stop words in a single textarea at `/malone/+admin`.

== Spelling and punctuation canonicalization ==

 * [[https://launchpad.net/products/launchpad/+bug/5417|#5417: Find accented forms when searching (e.g. Carlos Perelló Marín with "perello")]]

{o} When indexing and when searching, Launchpad should treat accented characters and other variants as their nearest non-variant equivalent -- treating ö as o, œ as oe, É as e, and so on. This lets people find words even if they do not know how to enter special letters. The mapping table should be tweaked over time to match what people are most likely to type when searching for particular characters. (This satisfies Ernie's use case.)

{o} Hyphens should be treated either as any punctuation, a space in a phrase, or nothing at all. For example, searching for `email` should return reports containing either ''email'' or ''e-mail'', and so should searching for `e-mail`. And searching for `gnome-app-install` should return reports containing ''gnome-app-install'', ''GNOME App Install'', or ''gnomeappinstall.xml'', but not a report containing the phrase ''can't install a GNOME app''.

{o} Similarly, Ascii apostrophes (`'`) and graves ({{{`}}} and {{{´}}}) should be treated either as any punctuation or nothing at all. For example, searching for `doesn´t` should return reports containing ''doesn't'', ''doesn{{{`}}}t'', ''doesn´t'', or ''doesnt'' -- and so should searching for `doesn't`, `doesn`{{{`}}}`t`, or `doesnt`. (This helps satisfy Marina's and Tarina's use cases.)

{o} All other punctuation (except for the special uses of `:`, `(`...`)`, `"`, `-`, and `+` described elsewhere in this specification) should be treated as a space in a phrase, and multiple spaces are collapsed. For example, searching for `kp_separator` should return results containing ''KP_Separator'', ''kp-separator'', or the phrase ''kp separator''. And searching for the phrase "`dialog spacing isnt really correct`" should return a bug report containing the phrase "`dialog spacing   isn't (really) correct`".

In all these cases, the search as entered should still be displayed verbatim in the search field, and in the results count on the results page.

== Ignoring stop words by default ==

{o} Launchpad should ignore common words like `a`, `the`, `in`, and `bug` when searching, unless they are prefixed with "`+`" or surrounded by quotes, or unless there are no non-stop words in the search string. For example, searching for `bug in the Job pane of the print dialog` might return ''37 results for '''Job pane print dialog'''''. As before, the search as entered should still be displayed verbatim in the search field, so it can easily be tweaked by the addition of `+` or `"` characters.

== Spelling suggestions ==

 * [[https://launchpad.net/products/malone/+bug/1264|#1264: Bug keyword search is overly-literal]]

{o} If one or more words in a search string are likely misspellings of other words, these should be presented as suggestions at the ends of the search results.

{o} If there are fewer than 20 results for the initial search, ''and'' one or more of the words are probably misspelled, ''and'' results exist for the most likely suggestion, those results should be displayed automatically underneath the results for the initial search, under the heading '''<''n''> results for ''<adjusted search>'':'''. For example, searching for `diaog` should automatically return results for ''dialog''.

== Presentation of search results ==

 * [[https://launchpad.net/products/malone/+bug/881|#881: Advanced search controls should not show on results pages]]
 * [[https://launchpad.net/products/malone/+bug/885|#885: Search form should say "Search Again" for non-canned searches]]
 * [[https://launchpad.net/products/malone/+bug/936|#936: Bug listing fails to use full screen width]]
 * [[https://launchpad.net/products/malone/+bug/3152|#3152: Provide prominent links from product bug listing to package bug listings]]
 * [[https://launchpad.net/products/malone/+bug/3752|#3752: Bug lists should provide a way to change the query]]
 * [[https://launchpad.net/products/malone/+bug/5542|#5542: Malone shouldn't say "No matching results found" (inaccurate and imprecise)]]

{o} The results for a search should appear like this, using the entire page width to minimize vertical scrolling without compromising the amount of information provided about each report.

{{attachment:results.jpg}}

{o} If a search has no results, the text "There are no bug reports matching ''search string''." should be printed, along with tips for tweaking the search, and the bottom navigation bar and search form should be omitted.

{o} The first line of a search result should consist of a bug icon representing the bug's importance in the current context (or highest importance value, if the bug is reported on multiple contexts), and a link to the bug report consisting of up to the first 80 characters of the bug's summary, ellipsized at the end if necessary.

{o} The second line should consist of whichever string of up to 80 characters from the bug's description includes the greatest possible number of matching search terms, with ties being broken by beginning at the start of the description if possible, otherwise by balancing context before and after, ideally using whole words. If the description does not include any search terms, the extract should begin at the start of the description. If the chosen extract does not start at the start of the description, it should be preceded by "...", and the same afterward if (as usual) the extract finishes before the end of the description.

{o} Strings in the summary or extract that match the search terms should be `<strong>`ly emphasized.

{o} The third line should consist of:
 * the bug number
 * links to the Bugs pages of up to two contexts (e.g. "in __Thunderbird__," "in __Thunderbird__ and __Firefox__," "in __Thunderbird__ and 2 others," or "in __Impilinux__ and 5 others,")
 * if there is a current context or a single context:
  * either "assigned to __Name Of Person__" or "not assigned"
  * "for __Milestone__," if it is targeted to a milestone, otherwise just ","
  * the status
 * if there is no current context and the bug has more than one context: "assigned in 3 of 5".

== Search help ==

 * [[http://launchpad.net/bugs/660283|#660283: Bug search pages should document valid search expressions]]

While this specification is being implemented, the operator implemented most recently should be advertised under the relevant search form. For example, when the ''product:'' operator is implemented, this should appear under the search form on distribution and package bug search pages:

 '''New:''' Use `-product:` to find only bugs that don't occur upstream. __More tips__

"More tips" should link to `https://launchpad.net/malone/help`, which should list all the search features implemented so far that are directly useful (this excludes bug number disambiguation, spelling suggestions etc), in chronological order.

{o} Once all of this specification (apart from this paragraph) has been fully implemented for two weeks, the hint should be removed, the "More tips" link should become "Search tips" alongside the search forms, and `/malone/help` should be rearranged into logical order.

== Bugs that are not addressed by this specification ==

 * [[https://launchpad.net/products/malone/+bug/41|#41: Buglist mode is not persistent]]
 * [[https://launchpad.net/products/malone/+bug/635|#635: Hard to ensure that a bug hasn't already been reported]]
 * [[https://launchpad.net/products/malone/+bug/996|#996: Help with duplicate finding]]
 * [[https://launchpad.net/products/malone/+bug/1331|#1331: Allow recording and use of canned searches]]
 * [[https://launchpad.net/products/malone/+bug/2753|#2753: Searching for "div" fails to find bug 1749]]
 * [[https://launchpad.net/products/malone/+bug/3881|#3881: Malone would be much more useful with a Gmail-like "Back to Search results"]]
 * [[https://launchpad.net/products/malone/+bug/4772|#4772: Allow advanced searching in FOAF (and elsewhere)]]
 * [[https://launchpad.net/products/malone/+bug/5394|#5394: Clicking on "Advanced search" should preserve simple search criteria]]
 * [[https://launchpad.net/products/malone/+bug/6708|#6708: would be nice to see reporter on bug list pages]]

== Unresolved issues ==

 * We are now implementing a second language accessing bug metadata, different to the one we are already using for setting it via the email interface. If the key:value syntax is becoming more familiar to people through google and its various tools, we should seriously consider changing the email interface to match. -- StuartBishop in [[https://launchpad.net/products/malone/+bug/5594|bug #5594]]
 * [[https://launchpad.net/products/malone/+bug/28697|#28697: bug lists should show current search filter]]
 * Is the `development` status shortcut value necessary?
 * [[https://launchpad.net/bugs/28503|ability to search a watch]]