Derivative Distributions

As an OEM provider
I want to make a customized version of Ubuntu
so that I can ship the OS best suited to my hardware

As a Linaro partner
I want to create a new distribution/archive in the hosting software in 3 or 4 clicks
so that this does not require deploying a new instance for each distribution.

As a Linaro partner
I want this new distribution visible only to a certain set of people, and only a subset of that group can upload
so that it's easy to see who is in each of those groups on a single page, at most 1 click from the overview page of the distribution. It is also simple to modify each of the groups, with at most 3 clicks required to add or remove someone from one of the groups.

As Meg who works for a partner enabling pre-release hardware
I want to upload modified packages to an archive for a hardware project
so that others who are working on the project can then build on top of this work.

As Lou, the lead developer on the project
I want to go to a web page and see the current state of the archive
so that I am able to remove some obsolete packages. This can be done from a single page, and is at most three clicks to remove a package. Bulk operation should be possible.

Rationale

We are doing this so it's easier for Linaro partners and Canonical's OEM team to build custom distributions.

This brings Launchpad's famous usability and feature set to more people who want to use it but currently have to use 3rd party products to do this.

Vocabulary

• A first initialization a the kind of initialization that happens when no other initialized series exists in the series' distribution. In this case, the packages are copied from the parents series and the differences (DSDs) are computed from these parents.

• Post-first initialization: This is the kind of initialization that happens when another initialized series exists in the same distribution. In this case, the packages are copied from the previous series and the differences are computed from the previous series' parents by default (but this can be changed on the +initseries page). All new Ubuntu series are post-first initialized series, parents is Debian Testing for LTS series and Debian Sid for non-LTS series.

Stakeholders

• Linaro
• OEM Services

Other blockers for stakeholder adoption

• OEM requires private distributions which is not covered in this LEP.

Constraints

• Ability to sync packages from the parent series, some auto, some manual, optionally rebuilding them
  • merge-o-matic type functionality, see merges.ubuntu.com
• Allow one distribution to have many derived distributions
  • These derived distributions must be discoverable from the "parent" distribution.
  • Once a distribution has a series that has derived from a series in another distribution, all other derived series must also derive from a series in the same distribution.
  • A distribution can have non-derived series. Any of these can be changed to derived at a later date, but as soon as this happens, the above rule applies.
  • It is permissible for a distribution to have both derived and non-derived series at the same time.
• Allow a derived distribution to have more than one parent
  • Post-initialisation, add more parents so that they appear in the list of differences

    • XXX - Can a distroseries be initialized from more than one parent? -- allenap

    • Not 'initialized', the intention is to add the other parents later and then sync from them so that you get all the conflict management shown on the diff pages.
• Track state of packages compared those in the parent's archive:
  • Report of differences against parent(s)
    • what might I want? what do I need to push upstream?
  • Action on reports: syncing packages in both directions
  • Sync all packages changed in parent but not changed in child
• Package branches for derived archives/distros
  • XXX - do we maintain official package branches in Launchpad or are they the responsibility of the derived distro "owner"? - jml
• Create a new distribution
  • Only XXX admins (commercial admins right now but this should change) can make new distributions, however the "driver" of the new distribution will be allowed to create/derive new series
• Create new derived distro series based on packagesets
  • Restricted to distro drivers
  • Required because people don't want to import all of parent (think about build time)
• Support different release cycles and release models from the parent distro
  • OEM case is frequently a small, finite number of rolling releases
• Open distroseries well before the series is ready for uploading so that QA folk can assign bugs to future distroseries
• Arbitrarily-named suites, with configurable policies

  • XXX - how are these created? who has permission to do so? how do they interact with bug affects & package branches? - jml

    • This means we need to remove the hard-coded rules around Ubuntu pockets and create a more generic mechanism to provide those rules around what can and cannot happen in different pockets. This should not affect "bug affects" or "package branches". I would envision Distro Drivers or Maintainers setting up the pockets, that needs to be clarified. -- julian

  • For now we can just add policies that mimic Ubuntu's and allow people to select which policies apply to the pockets/suites they create:
    • Freezing of the release pocket on release
    • Auto-overrides
    • XXX complete me

Nice to have

• new packagesets for derived distros based on parent's packagesets
• diff of packagesets
• User-defined periodic auto-sync of explicitly listed packages from parent
• Configurable constraints like:
  • preventing someone from uploading a package with the same version as a package in a parent distribution (binaries will have conflicting checksums)
  • uploading a package with an incorrect version (e.g. a version that will most likely be higher then the next upload in a parent distribution; ie. version scheme constraints).
  • XXX - "like"? Please list the configurable constraints we're providing.
  • XXX - What are the motivations for these features? - jml

Workflows

• Linaro partner creates a new distribution and derives a new series from an Ubuntu series.

• After a couple of weeks of hacking on a derived distribution, Meg syncs all the changes from Ubuntu into her derived series.

• Lou proposes a branch from his derived distribution to be merged into the upstream distribution

• Kiko looks at the Linaro distribution (derived from Ubuntu), gets a visual sense of the size of the delta between Ubuntu & Linaro. He investigates further, and sees that there are only one or two packages that are significantly different. He requests a merge of the Linaro packages to the upstream.

• Meg figures out that a bug in her derived distribution is actually a bug in the parent distribution. She files a bug and adds a bugtask for the upstream distribution package and a bugtask for her own package.

• Bryce is brought in to consult on an X problem in Linaro. He looks at the Linaro distribution and is able to quickly see the differences between Ubuntu & Linaro for X packageset packages only.

Success

How will we know when we are done?

Bugs are at: Bugs for this LEP

From https://wiki.ubuntu.com/Specs/M/ARMArchiveBranching: There will be a number of milestone test cases that we have:

• Having basic archive management working and able to accept uploads, build packages and publish them to an archive.
• Ability to branch a subset of Ubuntu in to the new archive management software.
• Ability to view differences between Ubuntu and a derivative using the web interface.
• Ability to do the same via the API.
• Ability to trigger syncs using the web interface.
• Ability to do the same via the API.

• https://wiki.ubuntu.com/NewReleaseCycleProcess can be done without shell access or manually contacting Launchpad developers. Broken down into:

  1. disable publisher via the API or the UI
  2. allow drivers to create new distroseries
  3. initialise package branches via API/UI
  4. 'Careful' publisher run to create new series indexes from scratch (for partner also)
  5. promote buildd chroots from previous series
  6. Everything else with seeds, overrides etc are all very Ubuntu-specific. Not sure we want to do that in LP at all.

How will we measure how well we have done?

Mockups

Creating a new distroseries

This is slightly simplified from the current one, and adds a help popup.

Initialising a distroseries

This is a separate step from the creation of distroseries since Ubuntu uses future uninitialised series as milestones.

New portlet for the distroseries page

This portlet can be replaced with a simple link to the initialisation page if it's currently uninitalised.

Questions for the derived distroseries portlet

• Be the first to add one.

Showing packages differences

Derived series differences

Questions for Derived series differences

• How can we communicate the changelog? Mathias had a good idea - make it easy for people to view the changelog without leaving the page. This is hard for a few reasons, but would be great to do in some form. So, as is, users can click on the Lucid package and then there's a link to the full changelog, and in the DB we have the most recent changelog entry.

  • Easiest incremental enhancement here would be to include the latest changelog entry in the alt text for the link... ie. it would only be discovered if someone went for more information (but even then, only if they hold the mouse still).
  • Next incremental enhancement would be a JS only one of adding a (view changelog) link next to the package via JS that displays the changelog in an overlay dialog (ie. without leaving the page).
  • Now that we're using a dropdown, we could display the latest changelog entry of either or both?

Derived series missing packages

Questions for Derived series missing packages

• Is the '6 new Lucid packages that have been ignored' link enough? When clicking on this link, all previously ignored packages will be displayed with an extra column for the comment. I'd rather do this than additionally include an extra filter option "include ignored packages" or similar? -- michael.nelson 2010-08-18 08:06:28

Derived series unique packages

Questions for Derived series unique packages

• Is there anything else useful for the table? (I've added description)

User testing

Round 1

Took place as remote audio calls on July 28th and August 2nd 2010, with:

• James Westby
• Mathias Gug
• Didier Roche.

Transcripts and audio
Recommendations

Round 2

This testing took place at UDS-O in Budapest on Friday 13th May 2011. It was implemented as a user or users driving the beta state of the software as running on the staging server where Oneiric was set up as a derivative of Sid. The following points are noted as a result of this testing:

Trivial UI

• Most users found the relevant new pages quite quickly. Martin Pitt expected to see the page at /ubuntu/oneiric/+source/<package> to contain information, however. It would be an interesting idea to add the expandable sections from the real diff pages to this page.

• Some users were confused by the "blacklisted" terminology, we should rename this to "ignored" 782215

• Colin didn't know what "resolved packages" meant, we should rename this to "resolved package differences" 782213

• The comments column has ellipsized text if it won't fit the column size, Colin would like to see the full text in a tool tip 782216

• The popup help text for "read more about syncing..." is mostly filler text. 782224

• The search box text says "show matching ..." which erroneously implies a glob/substring search is available.

• It would be useful to show the packageset that the package is in in the derived series. 783428

• Packagenames should be a green link to show it's a JS link. 783429

• Navigation links are missing at the bottom of the page. 783436

• The latest comment AJAX doesn't update the table column 746379

• We need a clickable link to the parent distroseries. 783441

• Make "last common version" a clickable link 783442

• The "update" button is useless, get rid of it. 783433

• There's a rogue line under the expanded sections, remove it. 783455

• Blacklist "this version" is not accurate enough, it should be "these versions" since the diff arises as a function of more than one version. 783430

UI

• The current searching facility (by full package name) is too restrictive, we also need to search by: (783423)

  • packagesets
  • people's names (uploader and maintainer)
  • ALL diffs, since we don't know if something is blacklisted (for example)
• Have in-page sorting

Features

• The archive admin's workflow is mostly driven by actions that they need to take, which basically means they are only really interested in differences where the parent distro's version is higher. We need to separate those from the others in the +localpackagediffs page. 784035

• The Ubuntu archive admins already maintain a flat file containing blacklisted packages that should never be synced. We need a way of importing that when this feature goes live.

• Higher versions in the child series are not syncable, yet we show a sync checkbox for it. 783435

• It would be "nice to have" a debdiff with just the debian/ directory in it.

Permissions

• The action of blacklisting should be restricted to archive admins. 782222

Bugs

• The aeskulap package was not showing links to request a diff

Thoughts?

• Check over the Ubuntu owner / bug supervisor etc model and see if it's relevant to non-standard projects
• Look up past literature on this subject
• What Ubuntu-specific scripts are we running against Launchpad?
  • XXX - how would you find out?
• Derivative distributions should be first class distributions.
• Configurable, entirely flexible, powerful policies about behaviors are key.

I see three main "features" here:

1. Creating distributions that can be uploaded to that will then build & publish packages.

2. Synchronizing distributions and the associated UI & new controls.

3. Creating derived distribution series.

There's also a bunch of associated stuff that must be considered:

• merging between branches that are associated via a derived distribution link
• searching for bugs etc. upstream, downstream and in "sibling" distributions

Notes on Ubuntu New Release Cycle Process

Conversation w/ jml & cjwatson at Platform Rally January 2011.

• According to cjwatson, Step 12, the need to do a careful publisher run is probably the top thing. If it were done, then probably the only thing left are the symlinks. https://bugs.launchpad.net/launchpad/+bug/55211

• i-f-p could be API'd then.
• Need to remove James Westby / Jonathan Lange from branch distro steps
• Need to API that

• more-extra-overrides is stuff that comes from outside the Launchpad database

  • task
  • supported fields
  • both come from germinate
• if you don't have the symlinks in place, then those fields will only get installed for main component packages.
• From LP pov, would be nice to have rosetta stuff integrated / automatic w/ initializing distroseries.
• API up 9, 10, 11, 12
• Possibly fixing permissions will smooth things out more. Fewer handoffs.

• https://bugs.launchpad.net/launchpad/+bugs?field.tag=new-release-cycle-process

• (No way to keep track of what was in a point release. Bug https://bugs.launchpad.net/launchpad/+bug/701595

Key steps from NRCP

These are steps 9-15 from https://wiki.ubuntu.com/NewReleaseCycleProcess

1. Create symbolic links /srv/launchpad.net/ubuntu-archive/ubuntu-misc/more-extra.override.DSN.restricted -> more-extra.override.DSN.main, /srv/launchpad.net/ubuntu-archive/ubuntu-misc/more-extra.override.DSN.universe -> more-extra.override.DSN.main, and /srv/launchpad.net/ubuntu-archive/ubuntu-misc/more-extra.override.DSN.multiverse -> more-extra.override.DSN.main on cocoplum, where DSN is the new distroseries name.

2. Notify Soyuz production team to run lp_publish:$ LPCONFIG=ftpmaster ./scripts/ftpmaster-tools/initialise-from-parent.py <new-distroseries-name> on cocoplum (takes around 8 minutes).

3. Notify James Westby/Jonathan Lange to initialize the branches for the new series.

4. Run the publisher once: lp_publish:$ LPCONFIG=ftpmaster-publish ./scripts/publish-distro.py -d ubuntu -vv -A -s DSN -s DSN-updates -s DSN-security -s DSN-proposed -s DSN-backports where DSN is the new distroseries name. This run will create the proper archive indexes for all suites (takes around 15 minutes).

5. As lp_archive, use compare-archives to compare dists trees for previous and current distroseries and sign off on any differences; the only differences should be the distroseries name, that custom uploads (installer-*, dist-upgrader-all, and i18n) are missing from dists/DSN/main, that Release.gpg does not yet exist (this will be created when the full publisher cron job next runs), and that Contents-*.gz do not yet exist (these will be created when generate-contents next runs).

6. Similarly, run the publisher once for the partner repository: lp_publish:$ LPCONFIG=ftpmaster-publish ./scripts/publish-distro.py -d ubuntu -vv -A -s DSN -s DSN-updates -s DSN-security -s DSN-proposed -s DSN-backports --partner -R /srv/launchpad.net/ubuntu-archive/ubuntu-partner/dists; compare and sign off on any differences. The new Packages and Sources files should be empty.

7. Re-enable the Soyuz publisher cron jobs and wait for the first run to complete.

Old mockups

Creating a new derived distroseries:

New portlet for the distroseries page:

Showing packages differences:

Random notes

• https://people.canonical.com/~timrc/omsk.png

• https://wiki.canonical.com/OEMServices/Engineering/StandardProjects/