Diff for "Soyuz/TechnicalDetails/UploadProcessor"

Not logged in - Log In / Register
Differences between revisions 1 and 2
⇤ ← Revision 1 as of 2011-09-09 11:58:16 →
Size: 732
Editor: julian-edwards
Comment:
← Revision 2 as of 2011-09-09 14:57:00 → ⇥
Size: 2194
Editor: julian-edwards
Comment:
Deletions are marked like this. Additions are marked like this.
Line 25: Line 25:

`process-upload.py` does many, many checks and then moves the files from the upload into:
 * accepted: if the upload passes the checks
 * failed: if there is a problem with the package itself
 * rejected: if the package is OK itself but it's rejected for some other reason, e.g. policy checks

The upload processor always sends email whether the package was accepted or rejected, except in the following circumstances:
 * The package's changes file is not properly GPG signed, or we can't match the GPG signature to a Launchpad person.
 * A bug that causes the upload processor to crash. The only known bug right now is that some exceptions are treated as a fatal error instead of an upload error, so they OOPS instead.

=== Ancestry and auto-acceptance ===

When a package is accepted, it can then go into one of three separate states in the PackageUpload table:
 1. if it is considered new, that is we've never seen that package before in the upload context, it is held in the distroseries NEW queue.
 1. if it's not new and has "ancestry" in the distroseries context, but the distroseries is either frozen or released, it is held in the UNAPPROVED queue
 1. if it's not new and doesn't meet the previous condition, it is "auto-accepted" - the PackageUpload moves to the DONE state and a PENDING publishing record is created.

If a package being uploaded is for a PPA then it is always auto-accepted. PPAs do not have distroseries queues.

Soyuz Upload Processing

Once package files are available on the upload servers, we need to run the Upload Processor to deal with them. This usually runs periodically from a cron job.

The Upload Processor

  • scripts/process-upload.py

  • lib/lp/archiveuploader/

The process-upload.py script is initiated with a single argument which is the base directory of the file upload area - let's call it $BASE. The catch is that this is the parent of $INCOMING because once things are processed from $INCOMING they are moved elsewhere.

The file structure looks like this: 

$BASE/
     |
     +-incoming/   ← $INCOMING
     |
     +-accepted/
     |
     +-failed/
     |
     +-rejected/

process-upload.py does many, many checks and then moves the files from the upload into:

  • accepted: if the upload passes the checks
  • failed: if there is a problem with the package itself
  • rejected: if the package is OK itself but it's rejected for some other reason, e.g. policy checks

The upload processor always sends email whether the package was accepted or rejected, except in the following circumstances:

  • The package's changes file is not properly GPG signed, or we can't match the GPG signature to a Launchpad person.
  • A bug that causes the upload processor to crash. The only known bug right now is that some exceptions are treated as a fatal error instead of an upload error, so they OOPS instead.

Ancestry and auto-acceptance

When a package is accepted, it can then go into one of three separate states in the PackageUpload table:

  1. if it is considered new, that is we've never seen that package before in the upload context, it is held in the distroseries NEW queue.
  2. if it's not new and has "ancestry" in the distroseries context, but the distroseries is either frozen or released, it is held in the UNAPPROVED queue

  3. if it's not new and doesn't meet the previous condition, it is "auto-accepted" - the PackageUpload moves to the DONE state and a PENDING publishing record is created.

If a package being uploaded is for a PPA then it is always auto-accepted. PPAs do not have distroseries queues.

Soyuz/TechnicalDetails/UploadProcessor (last edited 2011-09-12 10:46:17 by julian-edwards)