• Process Name: Release Manager Rotation Process

• Process Owner: Francis Lacoste

• Parent Process/Activity: None

• Supported Policy: None

Process Overview

Each cycle a different engineer takes the role of release manager. The release manager coordinates with the release team and all team leads to ensure that the tree is ready for the roll-out and that all critical bugs are in or worked-around.

Back-up release managers are the two RMs from the previous two cycles.

Release Manager inputs

• Email and IRC messages from engineers and team leads.
• OOPS report
• Merge proposal

Activities

Before the roll-out

• At the beginning of week 4. Make sure that release-critical was turned on in PQM.

• Update the #launchpad-dev topic to list him as release-manager.

• Maintain the list of the Current roll-out blockers

  • The release manager should poll the team leads and QA engineers continuously to ensure that the list of release blockers is up to date. All bugs that are likely to cause lots of OOPSes, time outs or prevent several users from working are good CRB candidates. It's a good idea to subscribe yourself to the page.
• Make sure that developers are assigned to all problems we want to fix.
• Review release-critical merge proposals.

On the day before the roll-out

• Request that landing to the devel branch be closed. (All changes should on the last day be merged through db-devel.)

On the day of the roll-out

• Chase up Current Rollout Blockers and any other pending release-critical fixes.

• Remind people that all changes need to be in buildbot for 6 hours before the roll-out time.

• In the case of failures, it's best to roll-out the last-known-good-build rather than delaying the release. The cut-off point to decide which revision

  to roll out is 2 hours before the scheduled release.

After the roll-out

• With the QA engineers, review the OOPS reports.
  • All common OOPSes are candidates for more release-critical fixes and scheduling another roll-out.
• Prepare and schedule any necessary re-roll.
• When a re-roll is needed, same activities than in the pre-roll out case.
• Open the tree, when the released version is fine for the next cycle.
• The release-manager need to select the next release manager.

Release critical policy

• To apply for a release-critical approval, you must have a reviewed merge proposal on Launchpad. The release manager simply adds a review of type

  release-critical to the merge proposal.

• Good candidates for release-critical approval are issues found during QA that are bound to create OOPSes and time outs or otherwise significantly inconvenience our end-users.
• Apart from special exceptions discussed with the project lead, only bug fixes should be granted release-critical approval.
• If there is no way for the developer to QA his change on staging through the normal update procedure before the roll-out, it's recommended to request a cowboy of the branch on staging to QA it before approval.
• For the second roll-out (a.k.a. the re-roll), any change requiring database changes should go through the project lead, since DB updates seriously increase the length of the upgrade window.

Scheduling

• Engineers apply in advance for one cycle.
• They are selected by the previous release manager. Once selected, their name

  is put on the Launchpad Production Status page.

• The actual roll-out time is determined by the release-manager's location:

  • Location	Roll out time
    Americas	00:00UTC
    Europe	09:00UTC
    Asia/Pacific	00:00UTC

• No engineer should apply for the role more than twice a year.