Rationale

We currently tie together two unrelated things:

• Changes to the database
• exposure of new features on 'launchpad.net'

This causes many problems for us, including:

• Unfinished features are deployed to the entire userbase.
• Finished features and small bugfixes/tweaks are held back from users for (on average) 2 weeks.
• We regularly encounter conflicts and test interactions between the 'released' and 'beta' code bases.
• We have a very complex deployment situation, with *at best* 2 versions of the code running at any one time, and regularly encounter friction between the two versions.

We want to decouple these two things, streamline and simplify.

Stakeholders

Launchpad developers: Mail sent to launchpad-dev. Developers have their processes impacted when we change development process, so need to be able to have their needs met.

Launchpad users: hard to have a discussion with all the users that are. affected. At least we give them a choice of using either edge.launchpad.net or launchpad.net. Users generally want Launchpad to be fast and reliable, and if we are successful with this LEP will get both of those things more often.

Constraints

• Allow a feature to exposed to edge.launchpad.net only, even if we rollout db changes to launchpad.net.

• Must be able to review the list of features that are exposed to less-than-every-user.

• Need to keep track of which revisions on edge haven't been pushed out to launchpad.net yet.

  • We are removing this distinction entirely.

• Must not slow down the current development process.

Out-of-scope

• Features that require database schema changes.

Nice-to-have

• Obvious when looking at edge that a feature is new and warrants feedback.

  • This isn't being delivered, though an idiom for doing it will be possible and fairly easy to do on a case by case basis.

Implementation

To detangle the two concerns (deployment and releasing) we need to have features in the code base enabled at runtime, rather than deployment time. This will be accomplished by LEP/FeatureFlags. Generally nothing should depend on the 'is_edge' check (other than branding for edge). The feature flags facility is now available in db-devel, and from 10.09 should be used for all changes which the developer does not want *immediately* given to users.

To enable more regular deployment (a necessary condition to avoid freezing up all releases) a number of deployment and code glitches need to be fixed. The following RT tickets are crucial to this LEP:

• 40477 XMLRPC server rolling upgrades
• 40480 codehosting rolling upgrades
• 40482 staging-with-production-schema environment
• 40685 deploy icing to apache before updating appservers

Some bugs in the LP codebase will also help, but are less strictly needed:

• http://pad.lv/607391

There may be other issues, but we will discover these if/when a deployment goes wrong, and feed them back into the process as high/critical bugs.

Lastly, a bug in edge - that we do not QA what we deploy - will be fixed as part of this process. The new workflow is described in MergeWorkflow.

Success

When we can update the db schema without rolling out features under. development, and the Launchpad developers haven't gone mad from crazy process changes.

Better quality features released to production.

Thoughts?