* '''Launchpad entry:''' https://launchpad.net/products/soyuz/+spec/native-source-syncing * '''Created:''' <> by DanielSilverstone * '''Contributors:''' CelsoProvidelo == Summary == The so called 'Native Source Sync' procedure will reduce the time spent by maintainers propagating uploads inside Soyuz. It will make it possible to instantly copy source and binary publications across suites or even distributions. == Rationale == While helping to reduce duplicated information stored in the SourcePackageRelease table when dealing with multiple distribution, Native Source Sync will also promote a quick and sane basis for "real world" derivation. For example, an updated package lives in the 'proposed' pocket until it is considered ready for release, at which point its version is incremented and it has to be re-uploaded to go to the 'updates' pocket. This is problematic because: * It creates duplicated source and binary package release files * The version has to change and the code is rebuilt which can introduce bugs == Use cases == * Colin is processing an update to dapper submitted by a developer, which has gone through testing in the PROPOSED pocket as specified in the [[http://wiki.ubuntu.com/StableReleaseUpdates|Ubuntu stable release updates process]]. He wishes to transfer this update directly into the UPDATES pocket without the need for a rebuild. * Scott wants to migrate a Debian source package to the current development Ubuntu release (currently handled by sync-source). * Fabio wants to include one of the packages published in his PPA to ubuntu/feisty without the need for a rebuild. * Raphael is a REVU member and does not have access to production system, but he wants to sync one of his PPA package to ubuntu/feisty. He can use either the restricted WEB UI or the XMLRPC client tool for it. == Assumptions == None. == Design == The NSS procedure will be entirely based on the current publishing system, no extra infrastructure parts will be required. It will be deployed in the following steps: 1. a command line tool on a machine with db access, limited to archive-admins 1. a web ui, that lets appropriate people say "move THIS to SOMEWHERE" 1. an xml-rpc version of the command-line client tool which can be given to a wider group of users Each request will be verified for sanity (see below) and submitted to the already existing queue processing cycle (NEW/UNAPPROVED, ACCEPTED/REJECTED, DONE). Sync uploads will be also subjected to the same rights that the requesters have for normal uploads. == Implementation == The implementation will be staged as follows: 1. copy between pockets (PROPOSED -> UPDATES, SECURITY -> UPDATES, no rebuild, i.e, include binaries) 1. copy between archives/PPAs ( cprov/ubuntu/feisty/RELEASE -> ubuntu/feisty/UPDATES) 1. copy between distrorelases (feisty-RELEASE -> edgy-BACKPORTS, rebuild, do not include binaries, replace backport-source.py) 1. copy between distribution (debian-unstable -> ubuntu.feisty-RELEASE, rebuild, do not include binaries, replace sync-source.py) For the stage 1 no extra features are required, we only need to extend the ChangeOverrideAndRemove API based on DistroReleaseSourcePackageRelease for sources and DistroArchReleaseBinaryPackage for binaries. Additionally to the ''changeOverride'' & ''supersede'' methods it will also implement a ''copyTo'' method as: {{{ def copyTo(distrorelease, pocket): """Copy this DistroReleaseSourcePackageRelease to another location. Return the publishing record in the targeted location. """ }}} During the stage 1 we will assert that the target_distrorelease is the same than the current published distrorelease, i.e., it will only work for copies between pockets. No DistroReleaseQueue background is required for operation in this stage. Distro-team is happy with the history kept in publishing tables. We will operate in COPY mode only in all stages, when necessary, the old (source) records should be explicitly removed using remove-package.py. After landing PersonalPackageArchives (PPA) we will need extend this implementation to support Archive model, stage 2, and probably go for stage 3 & 4 (which are very similar from the Launchpad point of view) The new pre-publication overrides system is only required for syncing from other distributions, specially Debian, in stage 4. Debian imports will only work as expected if we setup the gina to do incremental imports of Debian into Launchpad. Long story ... Also for stage 3 & 4, we will probably need be move some checks from NascentUpload pipeline to Queue pipeline, so they don't need to be duplicated but both, uploads and sync will be submitted to them. They are, for instance: * Claimed sources files match found SourcePackageRelease, * Claimed SourcePackageRelease matches claimed BinaryPackageRelease, * All BinaryPackageRelease generated by the found Build are included * ... == UI Changes == === Command-line script === The command-line tool should be based on LaunchpadScript infrastructure and respect the following format: {{{ $ copy-package.py [-d DISTRIBUTION] [-s SUITE] [-e SOURCEVERSION] [-b] <--to-suite SUITE> [--to-distribution DISTRIBUTION] -d DISTRIBUTION -> from-distribution (default: ubuntu) -s SUITE -> from-suite (default: DEVELOPMENT release) -e -> source version (default: current published) -b -> include or not binaries (default: false) --to-distribution -> to-distribution (default: ubuntu) --to-suite -> to-suite }}} === Launchpad Web UI === It will require a page on DistroReleaseSourcePackageRelease (ubuntu/feisty/+source/foo/1.0) allowed only for restricted group. Via that pages the user will be able to select: * Targeted suite name & archive given by a vocabulary * Target component & section (respecting the selected suite) * Whether or not the resulting binaries should be included * component, section and priority for each resulting binary. CelsoProvidelo: Pending mockup. === Launchpad XMLRPC script === Should be similar to the local command-line tool. == Code Changes == DistroReleaseSourcePackageRelease and DistroArchReleaseBinaryPackage will be modified to implements the ''copyTo'' method. CopyPackageHelper library/class will be created in order to allow comprehensive and testable procedure. (lib/canonical/launchpad/ftpmaster.py). Tests for this area will be in lib/canonical/launchpad/scripts/ftests/test_copypackage.py === Pre-Publication Overrides === Currently pre-publication overrides are done by modifying the original Component Section/Priority in the SPR/BPR record, which is wrong and wouldn't work properly for derivation. One solution would be create an new annotation table to store pre-publication overrides: {IDistroReleaseQueueSource, IDistroReleaseQueueBuild}.publish should be aware of the annotated overrides when creating a new SSPPH/SBPPH records == Schema Changes == DistroReleaseQueue.changesfile should be modified to allow empty values. This change will have implication in the SPR.changesfiles and related. Each BinaryPackageRelease generated by a binary upload would be overridden according to the following table. {{{ DRQBOverride( DRQB FK, BPR FK, component FK, section FK, priority dbschema, ) }}} Pre-publication source overrides can be collapsed in DistroreleaseQueueSource table if necessary: {{{ DRQSOverride( DRQS FK, SPR FK, component FK, section FK, ) }}} == Data Migration == No data migration is required, the new tables don't need populating. == Unresolved issues == CelsoProvidelo: the right place for the binary side of ChangeOverride/Remove/CopyTo API is DistroArchReleaseBinaryPackageRelease not DistroArchReleaseBinaryPackage. We may be take the advantage of touching those files and fix this problem. ---- CategoryProposal