Soyuz/NativeSourceSyncing

Not logged in - Log In / Register

Summary

The so called 'Native Source Sync' procedure will reduce the time spent by maintainers propagating uploads inside Soyuz.

It will make it possible to instantly copy source and binary publications across suites or even distributions.

Rationale

While helping to reduce duplicated information stored in the SourcePackageRelease table when dealing with multiple distribution, Native Source Sync will also promote a quick and sane basis for "real world" derivation.

For example, an updated package lives in the 'proposed' pocket until it is considered ready for release, at which point its version is incremented and it has to be re-uploaded to go to the 'updates' pocket. This is problematic because:

Use cases

Assumptions

None.

Design

The NSS procedure will be entirely based on the current publishing system, no extra infrastructure parts will be required.

It will be deployed in the following steps:

  1. a command line tool on a machine with db access, limited to archive-admins
  2. a web ui, that lets appropriate people say "move THIS to SOMEWHERE"
  3. an xml-rpc version of the command-line client tool which can be given to a wider group of users

Each request will be verified for sanity (see below) and submitted to the already existing queue processing cycle (NEW/UNAPPROVED, ACCEPTED/REJECTED, DONE). Sync uploads will be also subjected to the same rights that the requesters have for normal uploads.

Implementation

The implementation will be staged as follows:

  1. copy between pockets (PROPOSED -> UPDATES, SECURITY -> UPDATES, no rebuild, i.e, include binaries)

  2. copy between archives/PPAs ( cprov/ubuntu/feisty/RELEASE -> ubuntu/feisty/UPDATES)

  3. copy between distrorelases (feisty-RELEASE -> edgy-BACKPORTS, rebuild, do not include binaries, replace backport-source.py)

  4. copy between distribution (debian-unstable -> ubuntu.feisty-RELEASE, rebuild, do not include binaries, replace sync-source.py)

For the stage 1 no extra features are required, we only need to extend the ChangeOverrideAndRemove API based on DistroReleaseSourcePackageRelease for sources and DistroArchReleaseBinaryPackage for binaries. Additionally to the changeOverride & supersede methods it will also implement a copyTo method as: 

    def copyTo(distrorelease, pocket):                                            
        """Copy this DistroReleaseSourcePackageRelease to another location.          
                                                                                 
        Return the publishing record in the targeted location.                    
        """

During the stage 1 we will assert that the target_distrorelease is the same than the current published distrorelease, i.e., it will only work for copies between pockets.

No DistroReleaseQueue background is required for operation in this stage. Distro-team is happy with the history kept in publishing tables.

We will operate in COPY mode only in all stages, when necessary, the old (source) records should be explicitly removed using remove-package.py.

After landing PersonalPackageArchives (PPA) we will need extend this implementation to support Archive model, stage 2, and probably go for stage 3 & 4 (which are very similar from the Launchpad point of view)

The new pre-publication overrides system is only required for syncing from other distributions, specially Debian, in stage 4.

Debian imports will only work as expected if we setup the gina to do incremental imports of Debian into Launchpad. Long story ...

Also for stage 3 & 4, we will probably need be move some checks from NascentUpload pipeline to Queue pipeline, so they don't need to be duplicated but both, uploads and sync will be submitted to them. They are, for instance:

UI Changes

Command-line script

The command-line tool should be based on LaunchpadScript infrastructure and respect the following format: 

 $ copy-package.py [-d DISTRIBUTION] [-s SUITE] <SOURCENAME> [-e SOURCEVERSION] [-b] <--to-suite SUITE> [--to-distribution DISTRIBUTION]

-d DISTRIBUTION   -> from-distribution (default: ubuntu)

-s SUITE          -> from-suite (default: DEVELOPMENT release)

-e                -> source version (default:  current published)

-b                -> include or not binaries (default: false)

--to-distribution -> to-distribution (default: ubuntu)

--to-suite        -> to-suite

Launchpad Web UI

It will require a page on DistroReleaseSourcePackageRelease (ubuntu/feisty/+source/foo/1.0) allowed only for restricted group. Via that pages the user will be able to select:

CelsoProvidelo: Pending mockup.

Launchpad XMLRPC script

Should be similar to the local command-line tool.

Code Changes

Pre-Publication Overrides

Currently pre-publication overrides are done by modifying the original Component Section/Priority in the SPR/BPR record, which is wrong and wouldn't work properly for derivation. One solution would be create an new annotation table to store pre-publication overrides:

{IDistroReleaseQueueSource, IDistroReleaseQueueBuild}.publish should be aware of the annotated overrides when creating a new SSPPH/SBPPH records

Schema Changes

DistroReleaseQueue.changesfile should be modified to allow empty values. This change will have implication in the SPR.changesfiles and related.

Each BinaryPackageRelease generated by a binary upload would be overridden according to the following table. 

DRQBOverride(
   DRQB FK,
   BPR FK,
   component FK,
   section FK,
   priority dbschema,
   )

Pre-publication source overrides can be collapsed in DistroreleaseQueueSource table if necessary: 

DRQSOverride(
   DRQS FK,
   SPR FK,
   component FK,
   section FK,
   )

Data Migration

No data migration is required, the new tables don't need populating.

Unresolved issues

CelsoProvidelo: the right place for the binary side of ChangeOverride/Remove/CopyTo API is DistroArchReleaseBinaryPackageRelease not DistroArchReleaseBinaryPackage. We may be take the advantage of touching those files and fix this problem.

CategoryProposal

Soyuz/NativeSourceSyncing (last edited 2010-02-22 04:36:04 by mwhudson)