Archive index

As an Ubuntu packager
I want correct names, icons, and categories for applications to appear in Ubuntu Software Center automatically
so that I don’t need to remember to do it myself, or run the risk of messing it up.

As an application developer
I want to see the eventual application name, icon, category etc when the application is in my testing PPA
so that I can correct any errors before the application reaches one of the official repositories.

Ubuntu’s app-install-data-ubuntu and app-install-data-commercial packages, and for-purchase application metadata, should be replaced with a single automated system. Soyuz should produce for each archive it controls — including Multiverse, Canonical Partners, and every other PPA — an index of names, icons, summaries, categories, and keywords for all software items in the archive. (A “software item” in this sense mostly corresponds to a binary package, but in some cases one binary package contains multiple applications that should have separate information.) Launchpad should put this index in a standard place in the archive, and rebuild it whenever a package is added or changed.

Rationale

Since the beginning of Ubuntu’s Lucid cycle, we have wanted to get rid of app-install-data-ubuntu and app-install-data-commercial:

• they are slow and difficult to update

• the time we want to update app-install-data-ubuntu is when the archive is frozen

• there are exceptions and bugs, so software shows up in Ubuntu Software Center that isn’t installable (or conversely, is hidden as a “technical item” when it isn’t)
• they work only for Main, Universe, and Partner, not for PPAs, Multiverse, or third-party archives.

Ongoing cost of not doing this

• Whenever a graphical application is added to Main or Universe, or its icon changes, Michael Vogt needs to rebuild the app-install-data-ubuntu package. Almost every bug in this package represents a cost of not generating this index automatically.

• Whenever a for-purchase application is added to the Ubuntu Software Center store, Brian Thomason or Michael Vogt needs to manually add metadata for the package to the Software Center Agent. This can’t scale beyond a few dozen applications per week. (They also need to register the price of the application; that needs to be automated separately.)

• Whenever a graphical application is added to (or updated in) Canonical Partners, Brian Thomason has needed to remember to rebuild the app-install-data-commercial package. Almost every bug in that package represents a cost of not generating that index automatically. (In Natty, Canonical Partners will be merged with the for-purchase archive, but that will just change one kind of manual work to another.)

• Whenever an open-source application goes through the post-release process, the packager needs to add custom metadata fields to debian/control, metadata that duplicates existing fields in the application’s .desktop file. They shouldn’t have to do this.

Stakeholders

Matthew Paul Thomas, representing Michael Vogt and Brian Thomason

Implementation

File format

• Opaque to apt, so doesn't really matter
• RFC-822
• Localization of categories and keywords (in the translations file?)

Soyuz process

• Strip the data out of the package when building it, store it somewhere
• Publish the metadata file along with everything else

Roadmap

• Maybe start with non-localized data for PPAs, then localized in a future version?
• Maybe still have Ubuntu Software Center using appnstall-data instead for Main and Universe initially
  • iteratively survey the differences between app-install-data and the metadata Soyuz is producing
    • fix bugs in the packages and/or in Soyuz

actions:

• - client: write scripts to extract the needed data to LP - client: provide examples what the file

Issues

• Often a .desktop file is in a separate package from the package you're actually interested in
  • e.g. wesnoth-data vs. wesnoth
  • e.g. emacs-common contains the icon for emacs22
  • maybe this should be fixed in the packages themselves
• Debian may or may not be interested this
  • e.g. keeping packages and debtags in sync
• If bulk of metadata is not in Packages file then we can be nicer to Launchpad
• Filling the Librarian with icons is necessary but annoying
  • garbage-collect them when done?
• Current list view description translations should be migrated from app-install-data (currently ca. 4000 strings)
  • [long description translations come from DDTP data in the archive, but list view's short descriptions come from .desktop file's Comment field which is translated in app-install-data's Rosetta template]

Two parts:

• 1) LP export 2) apt support for downloading this

LP:

• what to export?
  • per arch, per pocket (main, universe)

  • one pkg -> multiple apps, app names are not uniq

  • desktop data parts
    • appname
    • packagename
    • Comment (friendly summary) - multi language
    • popcon / rating
    • keywords
    • iconname
    • Categories
    • mime-type
  • command not found data
    • per arch, per pocket (main, universe)

    • packagename -> binaries

    • PROBLEM diverts etc, real world problem
  • icons ?
    • uuencode?
    • gkticoncache ?
    • size?
    • format?
  • tagfile/rfc822 just like Packages
• how many files?
  • command-not-found
  • software-center
  • icons
• we need to hook into the build process to extract the desktop file, command not found file data, icons etc (problem is diverts)

Initial Implementation

• Put icons into repository
  • options: big tarball or gtk-icon-cache
• something needs to go once the build is finished and extract the .desktop file and icons, then export that metadata
  • setup another script once build is finished
• need to create a standard for what kind of metadata can be supplied within the package
  • debian/control modifications
  • debian/something.desktop with X-App-Install tags
  • debian/something.something for command-not-found hints
• Things this metadata might contain:
  • icons
  • package descriptions (also translated)
  • restart required
  • not screenshots/movies/sounds (handled elsewhere as not downloaded up-front in software center)
  • mimetype
  • hardware/software requirements (opengl etc)
  • whether it's available in my language (may change after package upload since translation packages are different)
    • similar problem to ratings updates
  • keywords and keyword translations
• we have information that can be extracted from a upload (icon)
• and changes that happen after a upload (rating)

Launchpad team requirements

The Launchpad team needs from the Ubuntu Software Center team:

• File format for the metadata
• TODO - software center team: The code to inspect Debian packages and extract the metadata

Success

We will we know we are done when the app-install-data-ubuntu and app-install-data-commercial packages are removed from the Ubuntu archive.