Launchpad Staging Server(s)
It is being rebuilt every day:
- At 00:10 UTC, emperor backs up its databases.
- When this is done (approximatly 00:25 UTC) staging.ubuntu.com is shutdown.
- The launchpad_staging database is then destroyed and restored from the newly created backup of the production launchpad database.
- A fresh copy of chinstrap:/home/warthogs/archives/rocketfuel-build/launchpad is made
- The database upgrade scripts are run to ensure the database schema patch level matches the requirements of the code patch level.
- The server is then restarted.
Instructions for doing a manual update are on StagingServerUpdate.
The output of this process is currently being emailed to email@example.com. Select the PostgreSQL topic to receive them.
Differences from production environment
- There is a special robots.txt file installed so the staging server does not get googled.
- No cron scripts are currently being run.
- Seperate the staging rebuild reports from the PostgreSQL ones and create a new launchpad-error-reports topic filter.
- cron scripts. We need to ensure they don't try to run during upgrades. We also need a single script to be run by con every 60 seconds instead of adding each one to cron individually (which will be a source of rollout errors).
Experimental vs. Staging
Currently staging is misnamed, as it is running bleeding edge code. We really need two servers - staging.ubuntu.com and experimental.ubuntu.com. Experimental will run bleeding edge code, and we may be able to allow developers to roll out branches that are not yet in rocketfuel. staging will be used for testing the next version for release. Most likely the process will be:
On Monday, the existing firstname.lastname@example.org/launchpad--staging branch is rolled out to production.
On Monday, email@example.com/launchpad--devel--0 is tagged as the new staging branch. This is rolled out to
- During the week, fixes are merged into the current staging branch to fix errors identified on the staging server.
This process means that new features will not land on production for at least 1 week. However, bugs identified on the staging server can be fixed. We have regular, scheduled, production updates.