Soyuz External Data

Soyuz currently uses pieces of data that are not stored inside Launchpad itself: "extra overrides" and the "package-arch-specific" file.

The former is used for adding Bugs:, Origin:, Supported: and Task: headers to the Packages files. The latter is used to override the architectures that packages specify to build on and is imported from Debian with some small local changes.

As a Launchpad user
I want to make sure all Soyuz external data is stored in Launchpad
so that tracking overrides in packages becomes easier.

As a user of the Derived Distributions feature
I want to configure a separate set of overrides for some packages in derived distros
so that I can set different custom overrides to Ubuntu's.

Rationale

Why are we doing this now? (Because it's better in than out.)

What value does this give our users? Which users?

• It allows all of LP's users to query overrides, instead of a few people with access to the archive machine.
  • This makes no sense as written, because the overrides are exported in the published archive. --cjwatson
• It allows users of Derived Distros to set separate overrides to Ubuntu's.

Stakeholders

Who really cares about this feature? When did you last talk to them?

• Mark Shuttleworth (emails exchanged 2nd week of September 2010)
• Linaro (for DDs)
• OEM (for DDs)

Constraints and Requirements

Must

We must ensure that any data used for the routine operation of Soyuz is stored inside of Launchpad.

Nice to have

• We could add extra functionality to query this information via the UI.
• Pre-parsing the p-a-s file and storing its representation in the database, rather than the text file itself.

Must not

Subfeatures

Workflows

The p-a-s file is stored in a Bazaar branch. This will not change, except it will not be checked out to disk, it will be parsed internally. Optionally, it can be pre-parsed to speed up the decisions made by the build system which currently parses it every time builds are created.

Extra overrides are generated by cron.germinate as part of the seed processing. This will change so that the seed sources are stored in Launchpad and processed as part of the publisher run. I'm not sure how much of https://launchpad.canonical.com/SeedManagementAndInheritance is still relevant, and this change almost certainly coincides with Packagesets too. More more needs to be done here to investigate.

• Seed sources are in Bazaar branches too, so if you're dealing with those internally for P-a-s then logically you ought to be happy to do so for seeds as well. With the new generate-extra-overrides script (lp:~cjwatson/launchpad/refactor-cron-germinate), this would just involve changing the seed_bases parameter when constructing SeedStructure instances, and then python-germinate will take care of checking them out internally (in future it ought to use bzrlib for this rather than a temporary checkout, but Launchpad shouldn't need to worry about the details of how it does this). However, right now, we take advantage of the default (http://people.canonical.com/~ubuntu-archive/seeds/) being under Ubuntu Engineering control to check out seeds from a few different places: for instance, Ubuntu is lp:~ubuntu-core-dev/ubuntu-seeds/ubuntu.precise while Kubuntu is lp:~kubuntu-dev/ubuntu-seeds/kubuntu.precise. Moving this into the database would probably involve attaching the equivalent of a list of (flavour, seed URL) to DistroSeries. There would be some value to this anyway as it would let us remove Ubuntu hardcoding from what remains of the cron.germinate script. --cjwatson

Success

How will we know when we are done?

We are done when:

• we can remove the externally checked-out package-arch-specific file, and use a parsed version that is stored inside LP's database.
• Extra overrides are maintained inside Launchpad, not in external files.

How will we measure how well we have done?

Thoughts?

Put everything else here. Better out than in.