QA Shepherd project developer notes

Design notes for the Shepherd project.

Algorithm details can be found on the QAProcessContinuousRollouts page.

User stories

• As an operator and developer, I want to read in the log files the detailed reasons that the shepherd took a particular action, so that debugging the tool is easier. (mars)

  • the log should record both the current state of the change sources and the state transitions that the shepherd executes based on said current state.

• As a user, I want to read an HTML report that tells me whether my branch has been promoted from QA to production, so I do not have to rely on yet more email to tell me what is going on. (Ursinha, gary)

  • We probably want to publish this on https://qa.launchpad.net

• As an operator, I want to run the parts of the rollout process individually and on-demand, so that resolving problems is easier. (mars)

  • This implies a small sharp script for each part - finding branches, doing promotion, reporting the system state.

• As an operator, I want to run any script without worrying if another copy of the script is already running and stomping on the data, so that on-demand runs and maintenance are safer and easier.

  • This implies a single-instance log, like a PID file.

  • As an operator, I want a loud warning if a PID file is more than X minutes old, so I have some foreknowledge that a hang or crash is blocking further updates from happening. (mars)
• As a user, I want to be warned once a day by email that my branch is ready for QA, so that I remember to do it. (mars, Ursinha)
• As a user, I want a message sent to a mailing list when a branch has been sitting in QA for more than X days, so that someone can roll back the branch and unblock updates. (mars, Ursinha)

  • This implies a QA policy with a grace period.

• As an operator, I want to toggle a switch that keeps the scripts from running, so that I can do maintenance and updates without hunting down and disabling a bunch of cron scripts. (mars)

  • Probably dropping a maintenance.txt file on disk would do this.

  • As a user, I want the log file and HTML reports to tell me when updates were aborted because a maintenance.txt is in place, so that I know why updates aren't happening. (mars)
  • As an operator, I want the log file and HTML reports to tell me how long ago the maintenance.txt file was created, so I know if someone forgot to remove the file by accident. (mars)
• As an operator, I would like the script to be deployed as the lpqateam user, so that I can find all of the parts of the QA process in the same place. (Ursinha, matsubara, gary)
• As an operator, I want a single push-button command to set maintenance mode, update the code, then remove maintenance mode again, so that I am less likely to make mistakes when deploying a new version. (mars)

  • Need both maintenance.txt and some way to wait for executing scripts to finish their work before proceeding.