launchpad developmentWe should move the package importer to be more part of Launchpad.
There are two parts to the package importer, the controller and the slaves.
Parts of the controller are similar to the job system, with the slaves being like jobs.
As a first step we will keep the slaves as API clients, and just run them from the job system.
We will have a cronscript which queries the job system for jobs to run and launches the slave jobs as necessary. We will hope to make this event-based when messaging is available.
The controller first has a thread to poll launchpad for new package publishings. We can convert this to use zope events.
When an event is received a job needs to be created that refers to the package. This job will then be picked up by the cronscript.
Open questions:
- Where and how to present the jobs?
Something like the secret (but public): https://code.launchpad.net/+code-imports/+machines
- Job priority (there now, but perhaps not needed)
- being able to requeue a package with "and do this one next" is pretty useful. It allows you to respond to a user immediately, rather than "and you'll get it once the 1000 entry queue flushes".
- Is it worth storing the extra data such as testament shas?
- Validity checking is nice to have, but currently all the cross-checking gets us is failing imports because we have to wipe the cache to say "yeah, this other place is more authoritative now".
- Is the self-checking when there are no jobs something that is worth doing?
- I would think you could easily just add a self-check-on-idle to the queue system.
