[[Soyuz/TechnicalDetails/Building|Next Page: Building]]

= Soyuz Upload Processing =

Once package files are available on the upload servers, we need to run the Upload Processor to deal with them.  This usually runs periodically from a cron job.

== The Upload Processor ==

 * `scripts/process-upload.py`
 * `lib/lp/archiveuploader/`

The process-upload.py script is initiated with a single argument which is the base directory of the file upload area - let's call it $BASE.  The catch is that this is the parent of $INCOMING because once things are processed from $INCOMING they are moved elsewhere.

The file structure looks like this:

{{{
$BASE/
     |
     +-incoming/   ← $INCOMING
     |
     +-accepted/
     |
     +-failed/
     |
     +-rejected/
}}}

`process-upload.py` does many, many checks and then moves the files from the upload into:
 * accepted: if the upload passes the checks
 * failed: if there is a problem with the package itself
 * rejected: if the package is OK itself but it's rejected for some other reason, e.g. policy checks

The upload processor always sends email whether the package was accepted or rejected, except in the following circumstances:
 * The package's changes file is not properly GPG signed, or we can't match the GPG signature to a Launchpad person.
 * A bug that causes the upload processor to crash.  The only known bug right now is that some exceptions are treated as a fatal error instead of an upload error, so they OOPS instead.

=== Ancestry and auto-acceptance ===

When a package is accepted, it can then go into one of three separate states in the `PackageUpload` table:
 1. if it is considered new, that is we've never seen that package before in the upload context, it is held in the distroseries NEW queue.
 1. if it's not new and has "ancestry" in the distroseries context, but the distroseries is either frozen or released, it is held in the UNAPPROVED queue
 1. if it's not new and doesn't meet the previous condition, it is "auto-accepted" - the `PackageUpload` moves to the DONE state and a PENDING publishing record is created.

If a package being uploaded is for a PPA then it is always auto-accepted.  PPAs do not have distroseries queues.

=== Dealing with packages in the distroseries queues ===

If a package is held in NEW or UNAPPROVED then an '''Archive Administrator''' needs to deal with it. These are people who have an appropriate entry in the `ArchivePermission` table for the QUEUE_ADMIN permission type.

There are two ways to view packages in the queue:
 1. The distroseries queue page, e.g. https://launchpad.net/ubuntu/oneiric/+queue
 1. `scripts/ftpmaster-tools/queue` - this is a script that needs to be run on the server machine. It is deprecated but still in active use because the UI page has some serious deficiencies and usability issues.

When the queue is being processed, the archive admin has three actions:
 1. Apply so-called ''overrides''. The archive admin can change the values of the ''component'' and the ''section'' for a package.
 1. Reject the upload - this moves it to the "REJECTED" queue.
 1. Accept the upload - this moves it to the "ACCEPTED" queue.

=== Processing accepted queue items ===
Packages auto-accepted at upload time never enter this state, there are basically only two types of upload that do:
 1. Binary uploads from builders.
 1. Packages that are manually accepted, as above.

The next stage is to process these accepted items, and there is a script `scripts/process-accepted.py` that does just this.  It has a few tasks:
 1. Close any bugs mentioned in the change log for the package.
 1. Create a PENDING publication.
 1. If it's a source, create a NEEDSBUILD build record and score the build appropriately.
 1. Change the status of the `PackageUpload` to DONE.

This script runs as the first part of the main publisher script, hourly for Ubuntu, and every 5 minutes for PPAs.

[[Soyuz/TechnicalDetails/Building|Next Page: Building]]