Soyuz/TechnicalDetails/UploadProcessor

Not logged in - Log In / Register

Next Page: Building

Soyuz Upload Processing

Once package files are available on the upload servers, we need to run the Upload Processor to deal with them. This usually runs periodically from a cron job.

The Upload Processor

The process-upload.py script is initiated with a single argument which is the base directory of the file upload area - let's call it $BASE. The catch is that this is the parent of $INCOMING because once things are processed from $INCOMING they are moved elsewhere.

The file structure looks like this:

$BASE/
     |
     +-incoming/   ← $INCOMING
     |
     +-accepted/
     |
     +-failed/
     |
     +-rejected/

process-upload.py does many, many checks and then moves the files from the upload into:

The upload processor always sends email whether the package was accepted or rejected, except in the following circumstances:

Ancestry and auto-acceptance

When a package is accepted, it can then go into one of three separate states in the PackageUpload table:

  1. if it is considered new, that is we've never seen that package before in the upload context, it is held in the distroseries NEW queue.
  2. if it's not new and has "ancestry" in the distroseries context, but the distroseries is either frozen or released, it is held in the UNAPPROVED queue
  3. if it's not new and doesn't meet the previous condition, it is "auto-accepted" - the PackageUpload moves to the DONE state and a PENDING publishing record is created.

If a package being uploaded is for a PPA then it is always auto-accepted. PPAs do not have distroseries queues.

Dealing with packages in the distroseries queues

If a package is held in NEW or UNAPPROVED then an Archive Administrator needs to deal with it. These are people who have an appropriate entry in the ArchivePermission table for the QUEUE_ADMIN permission type.

There are two ways to view packages in the queue:

  1. The distroseries queue page, e.g. https://launchpad.net/ubuntu/oneiric/+queue

  2. scripts/ftpmaster-tools/queue - this is a script that needs to be run on the server machine. It is deprecated but still in active use because the UI page has some serious deficiencies and usability issues.

When the queue is being processed, the archive admin has three actions:

  1. Apply so-called overrides. The archive admin can change the values of the component and the section for a package.

  2. Reject the upload - this moves it to the "REJECTED" queue.
  3. Accept the upload - this moves it to the "ACCEPTED" queue.

Processing accepted queue items

Packages auto-accepted at upload time never enter this state, there are basically only two types of upload that do:

  1. Binary uploads from builders.
  2. Packages that are manually accepted, as above.

The next stage is to process these accepted items, and there is a script scripts/process-accepted.py that does just this. It has a few tasks:

  1. Close any bugs mentioned in the change log for the package.
  2. Create a PENDING publication.
  3. If it's a source, create a NEEDSBUILD build record and score the build appropriately.
  4. Change the status of the PackageUpload to DONE.

This script runs as the first part of the main publisher script, hourly for Ubuntu, and every 5 minutes for PPAs.

Next Page: Building

Soyuz/TechnicalDetails/UploadProcessor (last edited 2011-09-12 10:46:17 by julian-edwards)