Status

lazr.jobrunner project at https://launchpad.net/lazr.jobrunner

Removal of job-running code from the Launchpad Tree: https://code.launchpad.net/~adeuring/launchpad/lp-lazr.jobrunner/+merge/97458

Merged

• Implementation of celery-based job running: https://code.launchpad.net/~abentley/lazr.jobrunner/run-via-celery

• Implementation of job-running code in lazr.jobrunner: https://code.launchpad.net/~adeuring/launchpad/lazr.jobrunner-more-tests

Aaron's Todo

• workers: one-time Launchpad init
• per job-type init
  • Possibly not needed, because not expensive.
  • Can be implemented in Job.run
• Resource limits
  • Maybe per-celeryd, since available resources doesn't change...
  • Oops if memory limit exceeded
  • Perhaps needs to be NIHed-- don't accept jobs while too much memory is in use.
• Fast lane/slow lane
• Upgrade story

Nice to have

• Timeline in oopses

Postponed

• Kill workers whose tasks error

Resources

• Main machine is ackee

• IRC log https://pastebin.canonical.com/62113/

• jjo suggests we use 1.5-2 cores and 1.5 G on ackee
  • therefore, probably one worker process each for fast and slow lanes
• lifeless revises this to 4 workers, suggesting 3 fast and 1 slow
• loganberry is also available, but heavily loaded

Open Questions

Currently, each job type has its own config. Do we retain that? If so, how do we associate jobs with their configs?

How do we support nodowntime/fastdowntime upgrades?

• Send SIGTERM to current celeryds and start new ones?
  • Less coding
  • more chance of exceeding resource limits as new workers take jobs while old workers are still working.
• On message or signal, workers stop accepting jobs, reschedule any running jobs, and exit?
  • More coding
  • No chance of exceeding resource limits
  • Delays slow-running jobs further

Analysis of current work load

Current work load

Fast-first vs slow-first

The fast lane must always time-out and dump timed-out jobs to the slow lane. But should jobs start in the fast lane or the slow lane?

Fast-first

If they start in the fast lane, then every slow job will run in the fast lane first, then be re-queued into the slow lane. If slow jobs take up every fast-lane worker, then latency will be introduced. Therefore, there competing incentives to maximize the number of fast workers (to avoid latency) and to maximize the number of slow workers (to increase the throughput of slow jobs).

Slow-first

Another way to organise this is to initially queue into the slow lane, and treat the fast lane as overflow when all slow-lane workers are occupied. In this case, latency is introduced only when all workers are busy, not all fast-lane workers. However, slow-lane workers are more likely to be busy than fast-lane workers, because they do not have a timeout. So while there is still an incentive to maximize the number of fast workers (to reduce the likelihood that all workers will be busy), it is less pronounced than with fast-first. Two fast-lane workers is probably plenty.

Analysis

While fast jobs are the norm (see Current work load), it's not clear whether slow jobs arrive in clusters. If they do, slow-first has a better chance to absorb the work without introducing latency.

Slow-first reduces the advantage of having many fast-lane workers, encouraging a more balanced set of workers. The fewer workers permitted, the more slow-first shows an advantage in latency and throughput.

lifeless rightly observes that slow-first is more complicated than fast-first, and also that slow-first can be built on fast-first. He has indicated a strong preference for fast-first, until the need for slow-first can be demonstrated.