• Process Name: Release Manager Rotation Process

• Process Owner: Francis Lacoste

• Parent Process/Activity: None

• Supported Policy: None

Process Overview

Each cycle a different engineer takes the role of release manager. The release manager coordinates with the release team and all team leads to ensure that the tree is ready for the roll-out and that all critical bugs are in or worked-around.

Back-up release managers are the two RMs from the previous two cycles.

One option that has worked very well is to share the release-manager role across timezones, handing over the current status and tasks to the backup in the next timezone at the end of your day. It's a great opportunity to work together as a team.

It is the incumbent release manager's job to find a willing person to manage the next release.

Release Manager inputs

• Email and IRC messages from engineers and team leads.
• OOPS reports
• Merge proposals

• LaunchpadProductionStatus (private)

• Deployment reports

  • stable

  • db-stable

Activities

When you start

• Negotiate and confirm the dates and times for the main release window and a backup release window with the following groups. Refer to the Launchpad Release Calendar for the pre-scheduled dates and times (we has a process in place to ensure our release calendar does not conflict with Ubuntu; conflicts normally do not happen). See Scheduling for more info on exact times. The goal is to confirm that the scheduled downtime does not interrupt other teams at critical times, to ensure that the release window does not place undue stress or risk upon our own operations team, and renegotiate the dates and times if necessary.

  • the Launchpad Project Lead

  • the LOSAs (losas@canonical.com)

  • OEM (solutions-group@lists.canonical.com)

  • the DBA (Stuart Bishop <stuart.bishop@canonical.com>)

  • Matthew Revell <matthew.revell@canonical.com>

  • and finally with:

    • the stakeholder's list (private-canonical-launchpad-stakeholders@lists.launchpad.net). jml on IRC can add you to the team.

    • the platform team (https://lists.canonical.com/mailman/listinfo/ubuntu-platform). msm on IRC can approve your subscription.

    • canonical-launchpad@lists.canonical.com

  • Check the stakeholders list archives for the mails I sent out if you are looking for a template. -- mars 2010-07-29 14:37:13

  • Consider when critical people have scheduled holidays and vacation around the release date. You can run a release without critical people like DB reviewers or the Project Lead, but it increases the stress and risk when something goes wrong. -- mars 2010-07-29 14:37:13

  • We consider the backup release window now because getting feedback from all the stakeholders can take a few days. Presumably you want to try a second rollout as soon as possible after the first (ask the LOSAs about what a reasonable backup window would be). -- mars 2010-07-29 14:52:37

End of Week 2

• Notify the canonical-launchpad list of the expected date and time for PQM closure. This should ideally happen at least 1 week before PQM closes; at the latest it should happen at the start of Week 3.

Week 3 (the one before the roll-out)

Preparation is required. Many items can be considered critical.

Start of Week 3

• Ensure that staging is up-to-date and had a DB restore so that it can be used for qastaging QA.

• The down-time estimate should be based on the last staging DB restore. We should be overly conservative in the read-only announcement. Recently, we always encountered DB locking issues during the roll-out which increased the read-only window. Until we are able to have stable DB upgrades and determine a reliable fudge factor between time on staging and production, we should plan for the worst and come back earlier.

• Follow the needs testing bugs and encourage engineers to QA before PQM closes. There are also deployment reports for stable and db-stable

During Week 3

• Ensure that Matt Revell has a downtime announcement email ready for lp-announce.

• Take part in the team leads call to coordinate the release. You should get invited but they might forget because this is new in 10.12.

• All QA for db changes must include checking that the aggregate database time is within budget. To find out how long the database updates will take to deploy on production, double the aggregate time from staging and add 10 minutes. Grep for lines like "2208-30-0 applied just now in 70.5 seconds"in devpad:/x/launchpad.net-logs/staging/sourcherry/2010-*-staging_restore.log

End of Week 3

On the day the PQM closes.

• Merge the last QA'ed revision of db-stable into db-devel
• Switch PQM to release-critical for devel.

• Update the #launchpad-dev topic to state we are in 'Release Critical' and to list the release manager.

• Ensure that qastaging will get a full, not nodowntime, update when the db-stable changes land in stable. (Talk to LOSAs)

• Also go through all the 'unusual rollout requirements' and clean them up from LaunchpadProductionStatus page

• Make sure all the cowboys listed on LaunchpadProductionStatus are either already landed or are added to the 'unusual rollout requirements' on that same page, so that the LOSAs will re-apply the cowboys after the rollout. It is definitely preferable not to re-apply cowboys, so it's a good idea to track down who hasn't landed a permanent fix yet.

• Offer to review anything that may need a release critical. Provide your RC to any work that may miss the PQM close time (EC2 is unpredictable). Providing RCs before the close of PQM allows engineers to work with some certainty that there work is required.

• Be prepared to make test fixes over the weekend to ensure the last branches are complete, or back them out so that staging is ready for testing Monday

Release Week

Start of Week 4

• At the beginning of week 4, schedule a call with the Foundations team lead (and other leads if known to be pertinent) to determine what system changes might need comprehensive QA. If these exist, consider these thoughts.
  • Any related problem encountered should be treated as a red flag, forcing more thorough QA.
  • Foundations lead should report on reviewing logs on edge, such as of cronscript output.
• Look at the staging DB restore time for the week-end and determine whether they will be within the 100-minute budget. If not, consult with the Launchpad team lead.

• Determine the schedule and deadlines. Send an email to launchpad-dev with all of the deadlines, similar to this example email. Place the deadlines on the team calendar.

• Check the qa report for stable. (You can ignore db-stable, since you only merged the revisions that had already been QAed.)

During Week 4

• The release manager should poll the team leads and QA engineers continuously to ensure that the release blockers in the stable qa report are being handled.

• Make sure that developers are assigned to all problems we want to fix.
• Review release-critical merge proposals. The policy should be:
  • All RC candidates go through the normal review process.
  • After code and UI review the MP is left in 'Needs Review' state.
  • A new review of type 'release-critical' is added to the MP and assigned to the release manager.
  • If the MP is approved for 'release-critical', the review is marked 'Approve' and the state of the MP is set to 'Approved'.
  • Avoid doing the code review yourself, because two reviews from the same person will confuse the landing scripts (ec2 land, lp-land).
• Once all revisions in stable are QA-ed, record stable's tip revno (this will be rolled out). The release manager (you!) has the final word on picking a revision for the release. Ask a losa to "prepare the branch" for rollout, and switch devel PQM back to normal landing mode.

On the day before the roll-out

• Pick a revision to release, if you haven't already done so. See above. It is critical that this be done today so that there is time for the remaining processes.
• Check that the LOSA do a staged deployment of the code. This means testing pushing and building the code to all the servers ahead of time. We are looking for any hidden build problems and to determine the amount of time this step will take.

On the day of the roll-out

• With PQM remaining open, have the LOSAs stop buildbot and set it do manual runs. This step just keeps the buildmaster process tree tidy when the codehost servers are restarted.

• Re-announce downtime so it serves as a reminder on launchpadstatus account at least 4h before the actual rollout. You can do this yourself with the identi.ca login info.

• Start a new RolloutReport to document any issues that will be encountered during the roll-out.

During the roll-out

• If some unexpected problems are encountered during the roll-out and these put the roll-out schedule off-track by more than 30 minutes, the decision to abort the roll-out should be taken. In that case:
• Identify the source of the problem and come up with a best estimate on how soon it can be fixed.
• Determine when the next attempt should be undertaken (usually: the backup window) and make sure it is announced. Give at least 12 hours of fore-notice.

After the roll-out

• Immediately after the roll-out, examine the site for problems. For example, ensure CSS loads properly, all external links on the front page are reachable, etc.

• Announce on launchpadstatus that the release is done.

• Announce any post-roll out issues on lauchpadstatus as they are discovered. Add them to the incident report. (Following the Launchpad/PolicyandProcess/Announcements/IncidentProcess, where the release manager acts as the communication liaison.)

• With the QA engineers, review the OOPS reports. All common OOPSes are candidates for cherry-picks.
• Complete the roll-out report and ask any engineers that participated in fixing the issues in doing root-cause analysis.
• Take part in the team leads call for a roll-out retrospective. The roll-out report will be the basis for that agenda item.
• The release-manager needs to select the next release manager.

Release critical policy

• To apply for a release-critical approval, you must have a reviewed merge proposal on Launchpad. The engineer adds a review of type

  release-critical to the merge proposal and ensures it is in the 'Needs Review' state.

• Good candidates for release-critical approval are issues found during QA that are bound to create OOPSes and time outs or otherwise significantly inconvenience our end-users.
• A good rule of thumb to apply is if this is something that would be likely

  to be deployed as a cherry-pick, it's good for release-critical.

• Apart from special exceptions discussed with the project lead, only bug fixes should be granted release-critical approval.
• If there is no way for the developer to QA his change on qastaging or staging through the normal update procedure before the roll-out, it's recommended to request a cowboy of the branch on staging to QA it before approval.

• If the current release manager isn't available for granting release-critical approval, you can seek approval using the regular Production Change Approval Policy. In this case, it is important that you ensure that the release manager is aware of that extra branch.

Database Patches

• Release-critical branches containing database patches should only be accepted if they don't impact the estimated roll-out time.
• One the day of the roll-out, only database patches that would be critical to prevent data corruption should be accepted.

Scheduling

• Engineers apply in advance for one cycle.
• They are selected by the previous release manager. Once selected, their name

  is put on the Launchpad Production Status page.

• The actual roll-out time is determined by the release-manager's location:

  • Location	Roll out time
    Americas	22:00UTC
    Europe	10:00UTC
    Asia/Pacific	22:00UTC

• No engineer should apply for the role more than twice a year.

References

• OSA Launchpad Rollout Procedures

• SpuriousFailures -- useful for diagnosing last-minute build failures

• QATeam/TestPlans -- proof from all the teams that their code works

• Launchpad production status

• Approval of production change

• Emergency change (aka cherry-picks)