Diff for "ArchiveIndex"

Not logged in - Log In / Register

Differences between revisions 4 and 5
Revision 4 as of 2010-11-02 14:37:28
Size: 5580
Editor: mpt
Comment: actually, that wasn't necessary
Revision 5 as of 2010-11-02 15:09:09
Size: 7639
Editor: mpt
Comment: updated to follow the LEP format; + "Ongoing cost of not doing this"
Deletions are marked like this. Additions are marked like this.
Line 5: Line 5:
We want to replace `app-install-data` and related packages with an automated system. By March 2010, Launchpad should be producing for every archive it controls — including Multiverse, Canonical Partner, and every PPA — an index of names, icons, summaries, categories, and keywords for all software items in the archive. A “software item” in this sense mostly corresponds to a binary package, but in some cases one binary package contains multiple applications that should have separate information. Launchpad should put this index in a standard place in the archive, and rebuild it whenever a package is added or changed. '''As an''' Ubuntu packager<<BR>>
'''I want''' correct names, icons, and categories for applications to appear in Ubuntu Software Center automatically<<BR>>
'''so that''' I don’t need to remember to do it myself, or run the risk of messing it up.

'''As an''' application developer<<BR>>
'''I want''' to see the eventual application name, icon, category etc when the application is in my testing PPA<<BR>>
'''so that''' I can correct any errors before the application reaches one of the official repositories.

Ubuntu’s `app-install-data-ubuntu` and `app-install-data-commercial` packages, and for-purchase application metadata, should be replaced with a single automated system. Soyuz should produce for each archive it controls — including Multiverse, Canonical Partner, and every PPA — an index of names, icons, summaries, categories, and keywords for all software items in the archive. (A “software item” in this sense mostly corresponds to a binary package, but in some cases one binary package contains multiple applications that should have separate information.) Launchpad should put this index in a standard place in the archive, and rebuild it whenever a package is added or changed.
Line 9: Line 17:
`app-install-data` is something we want to get rid of:
 * it's slow and manual to update
 * the time we want to update this package is when the archive is frozen
 * there are exceptions and bugs, so software shows up in Ubuntu Software Center that isn't installable
 * it's difficult to update the data
 * it works only for Main and Universe, not for PPAs or other archives
Since the beginning of Ubuntu’s Lucid cycle, we have wanted to get rid of `app-install-data-ubuntu` and `app-install-data-commercial`:
 * they are slow and difficult to update
 * the time we want to update `app-install-data-ubuntu` is when the archive is frozen
 * there are exceptions and bugs, so software shows up in Ubuntu Software Center that isn’t installable
 * they work only for Main, Universe, and Partner not for PPAs, Multiverse, or third-party archives.
Line 16: Line 23:
== User stories == == Ongoing cost of not doing this ==
Line 18: Line 25:
 * Maree maintains 16 packages in a PPA. She wants these packages to show up, with proper names, icons, departments etc, in the Ubuntu Software Center for anyone who adds her PPA.  * Whenever a graphical application is added to Main or Universe, or its icon changes, Michael Vogt needs to rebuild the `app-install-data-ubuntu` package. Almost [[https://bugs.launchpad.net/ubuntu/+source/app-install-data-ubuntu/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status:list=NEW&field.status:list=OPINION&field.status:list=EXPIRED&field.status:list=CONFIRMED&field.status:list=TRIAGED&field.status:list=INPROGRESS&field.status:list=FIXCOMMITTED&field.status:list=FIXRELEASED|every bug in this package]] represents a cost of not generating this index automatically.
Line 20: Line 27:
 * Brian has packaged a new version of Adobe Reader and published it to the Canonical partner repository. It has a different icon from the previous version.  * Whenever a graphical application is added to (or updated in) Canonical Partners, Brian Thomason needs to remember to rebuild the `app-install-data-commercial` package. Almost [[https://bugs.launchpad.net/ubuntu/+source/app-install-data-commercial/+bugs?field.searchtext=&orderby=-importance&search=Search&field.status:list=NEW&field.status:list=OPINION&field.status:list=EXPIRED&field.status:list=CONFIRMED&field.status:list=TRIAGED&field.status:list=INPROGRESS&field.status:list=FIXCOMMITTED&field.status:list=FIXRELEASED|every bug in that package]] represents a cost of not generating that index automatically.
Line 22: Line 29:
 * Ben uses apt-get to install everything. He isn't interested in downloading icons etc for applications he is never going to install.  * Whenever a for-purchase application is added to the Ubuntu Software Center store, Brian Thomason or Michael Vogt needs to manually add metadata for the package to the Software Center Agent. This can’t scale beyond a few dozen applications per week. (They also need to register the price of the application; that needs to be automated separately.)

 * Whenever an open-source application goes through the post-release process, the packager needs to [[https://wiki.ubuntu.com/PostReleaseApps/Metadata|add custom metadata fields]] to `debian/control`, metadata that duplicates existing fields in the application’s .desktop file. They shouldn’t have to do this.
Line 127: Line 136:
Needs doing by beta 1, or not in Lucid at all.

Archive index

As an Ubuntu packager
I want correct names, icons, and categories for applications to appear in Ubuntu Software Center automatically
so that I don’t need to remember to do it myself, or run the risk of messing it up.

As an application developer
I want to see the eventual application name, icon, category etc when the application is in my testing PPA
so that I can correct any errors before the application reaches one of the official repositories.

Ubuntu’s app-install-data-ubuntu and app-install-data-commercial packages, and for-purchase application metadata, should be replaced with a single automated system. Soyuz should produce for each archive it controls — including Multiverse, Canonical Partner, and every PPA — an index of names, icons, summaries, categories, and keywords for all software items in the archive. (A “software item” in this sense mostly corresponds to a binary package, but in some cases one binary package contains multiple applications that should have separate information.) Launchpad should put this index in a standard place in the archive, and rebuild it whenever a package is added or changed.

Rationale

Since the beginning of Ubuntu’s Lucid cycle, we have wanted to get rid of app-install-data-ubuntu and app-install-data-commercial:

  • they are slow and difficult to update
  • the time we want to update app-install-data-ubuntu is when the archive is frozen

  • there are exceptions and bugs, so software shows up in Ubuntu Software Center that isn’t installable
  • they work only for Main, Universe, and Partner not for PPAs, Multiverse, or third-party archives.

Ongoing cost of not doing this

  • Whenever a graphical application is added to Main or Universe, or its icon changes, Michael Vogt needs to rebuild the app-install-data-ubuntu package. Almost every bug in this package represents a cost of not generating this index automatically.

  • Whenever a graphical application is added to (or updated in) Canonical Partners, Brian Thomason needs to remember to rebuild the app-install-data-commercial package. Almost every bug in that package represents a cost of not generating that index automatically.

  • Whenever a for-purchase application is added to the Ubuntu Software Center store, Brian Thomason or Michael Vogt needs to manually add metadata for the package to the Software Center Agent. This can’t scale beyond a few dozen applications per week. (They also need to register the price of the application; that needs to be automated separately.)
  • Whenever an open-source application goes through the post-release process, the packager needs to add custom metadata fields to debian/control, metadata that duplicates existing fields in the application’s .desktop file. They shouldn’t have to do this.

File format

  • Opaque to apt, so doesn't really matter
  • RFC-822
  • Localization of categories and keywords (in the translations file?)

Soyuz process

  • Strip the data out of the package when building it, store it somewhere
  • Publish the metadata file along with everything else

Roadmap

  • Maybe start with non-localized data for PPAs in Lucid, then localized in a future version?
  • Maybe still have Ubuntu Software Center using appnstall-data instead for Main and Universe in Lucid
    • iteratively survey the differences between app-install-data and the metadata Soyuz is producing
      • fix bugs in the packages and/or in Soyuz

actions:

  • - client: write scripts to extract the needed data to LP - client: provide examples what the file

Issues

  • Often a .desktop file is in a separate package from the package you're actually interested in
    • e.g. wesnoth-data vs. wesnoth
    • e.g. emacs-common contains the icon for emacs22
    • maybe this should be fixed in the packages themselves
  • Debian may or may not be interested this
    • e.g. keeping packages and debtags in sync
  • If bulk of metadata is not in Packages file then we can be nicer to Launchpad
  • Filling the Librarian with icons is necessary but annoying
    • garbage-collect them when done?
  • Current list view description translations should be migrated from app-install-data (currently ca. 4000 strings)
    • [long description translations come from DDTP data in the archive, but list view's short descriptions come from .desktop file's Comment field which is translated in app-install-data's Rosetta template]

Two parts:

  • 1) LP export 2) apt support for downloading this

LP:

  • what to export?
    • per arch, per pocket (main, universe)
    • one pkg -> multiple apps, app names are not uniq

    • desktop data parts
      • appname
      • packagename
      • Comment (friendly summary) - multi language
      • popcon / rating
      • keywords
      • iconname
      • Categories
      • mime-type
    • command not found data
      • per arch, per pocket (main, universe)
      • packagename -> binaries

      • PROBLEM diverts etc, real world problem
    • icons ?
      • uuencode?
      • gkticoncache ?
      • size?
      • format?
    • tagfile/rfc822 just like Packages
  • how many files?
    • command-not-found
    • software-center
    • icons
  • we need to hook into the build process to extract the desktop file, command not found file data, icons etc (problem is diverts)

Initial Implementation

  • Put icons into repository
    • options: big tarball or gtk-icon-cache
  • something needs to go once the build is finished and extract the .desktop file and icons, then export that metadata
    • setup another script once build is finished
  • need to create a standard for what kind of metadata can be supplied within the package
    • debian/control modifications
    • debian/something.desktop with X-App-Install tags
    • debian/something.something for command-not-found hints
  • Things this metadata might contain:
    • icons
    • package descriptions (also translated)
    • restart required
    • not screenshots/movies/sounds (handled elsewhere as not downloaded up-front in software center)
    • mimetype
    • hardware/software requirements (opengl etc)
    • whether it's available in my language (may change after package upload since translation packages are different)
      • similar problem to ratings updates
    • keywords and keyword translations
  • we have information that can be extracted from a upload (icon)
  • and changes that happen after a upload (rating)

Launchpad team requirements

The Launchpad team needs from the Ubuntu Software Center team:

  • File format for the metadata
  • TODO - software center team: The code to inspect Debian packages and extract the metadata

ArchiveIndex (last edited 2010-11-04 10:31:26 by mpt)