Launchpad runs a continuous beta program, as well as special beta test campaigns. Access to the beta program is governed by membership in the launchpad-beta-testers team. There are around 1000 beta testers and we encourage enthusiastic users to join that team to get the very latest functionality in Launchpad.
Some major features *require* a beta test phase, but mostly the beta testing is useful for gaining feedback and for creating excitement around new work that has been delivered.
There is a user facing page about this at https://help.launchpad.net/BetaTesting
Continuous betas
All Launchpad servers run the latest QA'd code from the Launchpad developers. We have a facility to enable new code paths on a per-team basis. Significant new work that is ready for beta testing is turned on just for the developers and the beta test steam.
Special beta campaigns
We also run regular special beta campaigns, where we ask our beta testers to try a specific feature that is in beta, and give us feedback, before we make that feature available to the wider user base. We aim to have one feature in special beta testing at any point.
Rationale for the beta program
The beta program brings us:
Good testing of new features against real world data, and subject to real-world patterns of usage. We are able to identify system crashes, performance problems and user interface issues with new code before it is being hammered by the full community of Launchpad testers.
Passionate and helpful users - folks who enjoy Launchpad and are interested in the latest developments are also people that will advocate the platform, help provide guidance, suggest new features, and help other users when they run into problems.
Passionate users are essential!
Probably the most important element of the beta program is that it is a team of nearly 1,000 people who have been interested enough in Launchpad to want to sign up to the team, receive some email and chat with others. We really should interact more directly with this team, it will be fantastic to have a mailing list for the team so folks who are *really* interested can sign up to that.
People do not stay passionate if they don't have active conversations with other passionate users. They want to feed this habit, they want to participate, they want to be drivers and conductors as much as passengers in this race. The beta program is a GREAT opportunity to make our best users feel like they are in the drivers seat, specifically with regard to features we HAVE ALREADY IMPLEMENTED. The great thing about the beta program is that the discussion is not about something we might schedule to start work on in six months, it's about something that is ABOUT to be revealed to the whole Launchpad audience. So ideas, tweaks, innovations, improvements are really timely.
We need to treat the beta team with some just-for-you surprises now and then. Every month, we want to have a special campaign of code that is on edge.launchpad.net but only visible to beta testers. Read http://headrush.typepad.com/creating_passionate_users/2005/05/the_case_for_ea.html if you haven't already - its all about giving your best users something special, something just for them.
Beta campaign best practices
So, down to the nuts and bolts. This is what it takes to run a really good special beta campaign. Every developer in Launchpad is entitled to have a feature they are working on get a special beta campaign. Consider it a privilege!
One feature at a time - we will only have a campaign for a single feature at a time. That means we have a laser-like focus with our beta team on a specific thing we are landing. It means that the email which goes out to them can focus on one and only one thing we want to discuss and test. A campaign can be as short as a week or as long as a full release cycle - we can run multiple campaigns in a single cycle if we are well prepared.
Personalized invitations to test the feature are sent to the beta testers when we start a special beta campaign. The mail can be sent by the lead developer, or someone else if that developer does not want to do so. The requirement is that whoever sends the mail replies to *every* response they get - from experience that's 20-50 emails.
The code is testable immediately by beta testers. The feature should be right there on edge, your email to the beta team members will point them to a few examples of people who have used it already (i.e. you
and it will also include the link where they can try it out immediately themselves. DO NOT run a beta program off staging, or off dogfood, or off demo, or any other special program. By all means, get Canonical folks to test your code in those places, but don't announce that or consider it a beta program. It is not a beta program if it is not in production and instantly accessible. Exclusivity to beta testers. The beta team is special - they get something that nobody else has, which is access to exclusive features before widespread release. If we make the features generally available, we take all the fun out of the beta program (and we risk having a feature flooded before it is ready).
Implementing beta campaigns
It's easy to run a beta campaign!
Land your database changes early, in the cycle before you plan to run the campaign. This gives you freedom and flexibility to start the beta program when you want. Your database changes can even be quite intrusive, just preserve the old behaviour. For example, loc.diff is a change that removed Person.timezone, moving it to a new table, while preserving the old read and write semantics of Person.timezone while the new functionality could be developed.
Discuss the beta campaign with the product team - we want to make sure your beta campaign gets the best reception and fits in with the schedule of other announcements going on at Canonical. We'll agree how long the campaign is likely to be, and whether there are other things we can do to draw attention to the feature.
Make access to the features, menus, portlets, forms and pages conditional on a feature flag. You can easily do this. <tal:featurefoo condition="request/features/example.feature.enabled">. In python code, lp.services.features.getFeatureFlag("example.feature.enabled").
Your tests will need to enable the flag. Use the FeatureFixture fixture. When you "pull the curtain away" we just set the flag to enabled for the default scope and subsequently remove the checking of the feature in the code base.
When ready, make an announcement. You can use the spam.py script to send your announcement to the beta testers. You'll need to get a LP admin to run a SQL query against the staging database. You can find everything you need in this branch: bzr+ssh://devpad.canonical.com/code/barry/massmail Also see Sending Mass Email
Good luck, enjoy it!