Celery Jobs

Celery Jobs are used in production, however the mechanism for auto-fallback between 'fast' and 'slow' lanes is currently broken. Long running jobs may hit the 300 second timeout and fail. All active job types have been migrated to support running under Celery.

Data flow

The database remains the authoritative source of information about a job. Celery is merely a mechanism for running that job. Each attempt to run a job will use its own Celery task. As that task starts, runs, and terminates the job, the status and lease fields in the database are updated as normal.

This may change in the future, because Celery's Task class and Launchpad's Job class are very similar. They could merge, or one could replace the other.

Additional Requirements

Job factory methods should call Job.celeryRunOnCommit(). This schedules the job to run if/when the current transaction is committed. It only affects jobs whose class names are in the jobs.celery.enabled_classes feature flag.

To ensure that jobs cannot fall through the cracks, job sources for jobs that are (or are expected to be) run via celery should be added to lp.services.job.celeryjob.run_missing_ready()

Jobs must provide a 'config' member, which is meant to be a config section, e.g. lp.services.config.config.branchscanner. This member must have a dbuser attribute. The dbuser for each job type should be unique, to aid debugging production problems.

Database job classes such as BranchJob must provide a 'makeDerived' member which returns an IRunnableJob implementation. If there are no derived classes, as with POFileStatsJob, it should simply return itself.

Base classes for specific IRunnableJob classes based on an enum, such as BranchJobDerived, should use EnumeratedSubclass as their metaclass. This will cause each of their subclasses to be registered with the base class when that subclass is declared. EnumeratedSubclass automatically provides the makeSubclass method, which returns an instance of the appropriate subclass for a given database class. This allows makeDerived to be implemented as a one-liner:

    def makeDerived(self):
        return BranchJobDerived.makeSubclass(self)

It is not clear that FooDerived classes are needed, because Storm appears to permit one table to be associated with many classes. Further testing is required.

Job classes should provide a 'context' member which is a database class.

Jobs which cannot tolerate parallelism should use lp.services.database.locking.try_advisory_lock to avoid it.

Testing

Jobs should have an integration test to show that they run under Celery correctly.

Here is an example:

class TestViaCelery(TestCaseWithFactory):

    layer = CeleryJobLayer

    def test_DerivedDistroseriesDifferenceJob(self):
        self.useFixture(FeatureFixture({
            FEATURE_FLAG_ENABLE_MODULE: u'on',
            'jobs.celery.enabled_classes': 'DistroSeriesDifferenceJob',
            }))
        dsp = self.factory.makeDistroSeriesParent()
        package = self.factory.makeSourcePackageName()
        with block_on_job():
            job = create_job(dsp.derived_series, package, dsp.parent_series)
            transaction.commit()
        self.assertEqual(JobStatus.COMPLETED, job.status)

The CeleryJobLayer ensures a celeryd is running, to process the job. Configuring the feature flag jobs.celery.enabled_classes='DistroSeriesDifferenceJob' ensures that create_job will request the job to run under celery (as a commit hook). transaction.commit() causes the commit hook to fire, requesting the job to run via celery. lp.services.job.tests.block_on_job waits until the job that was requested to run via celery in its scope has completed.

Running celery in development

./utilities/manage-celery-workers.sh start will start celery with the specified concurrency. There is a set of workers per queue, multiple queues per worker is disallowed. Logs are in /var/tmp/celery_<queue_name>.log. You will also need to enable the job under the feature flag jobs.celery.enabled_classes.

QA

celeryd is running on gandwana, as staging. To test your code, it must land in the db-stable branch, and you can enable the job using the jobs.celery.enabled_classes feature flag. Dogfood (labbu) runs celery in development configuration using start-stop-daemon with the same concurrency configuration as production. All queues are consumed.

Production

celeryd is running on ackee. There is at least 1 worker for all queues listed in the configuration.

launchpad development

CeleryJobs