launchpad developmentLaunchpad runs a continuous beta program, as well as special beta test campaigns. Access to the beta program is governed by membership in the launchpad-beta-testers team. There are around 1000 beta testers and we encourage enthusiastic users to join that team to get the very latest functionality in Launchpad.
Some major features *require* a beta test phase, but mostly the beta testing is useful for gaining feedback and for creating excitement around new work that has been delivered.
There is a public page about this at https://help.launchpad.net/BetaTesting
Continuous "edge" betas
The "edge" servers in Launchpad generally run the latest daily code from the Launchpad developers, against the production database (real live data). People can access those servers by prefixing launchpad.net with "edge.", for example "bugs.edge.launchpad.net". If people pass out an edge URL to folks who are not beta testers, they are redirected to the non-beta servers, so by and large the URL's are interchangeable. Beta testers who go to the normal launchpad.net production servers will also be redirected to the edge servers, unless they have turned that redirection off for a period of time, which they can do at the home page, https://launchpad.net/.
The edge servers are production servers in the sense that any modifications or updates made to your data in Launchpad through edge.launchpad.net is affecting the same production database that everybody else is using on plain launchpad.net. Thanks to Bazaar, the updates to the edge application servers are all automated and happen daily.
Special beta campaigns
We also run regular special beta campaigns, where we ask our beta testers to try a specific feature that is in beta, and give us feedback, before we make that feature available to the wider user base. We aim to have one feature in special beta testing in each release cycle.
Rationale for the beta program
The beta program brings us:
Good testing of the latest daily code against real world data, and subject to real-world patterns of usage. We are able to identify system crashes, performance problems and user interface issues with new code before it is being hammered by the full community of Launchpad testers.
Passionate and helpful users - folks who enjoy Launchpad and are interested in the latest developments are also people that will advocate the platform, help provide guidance, suggest new features, and help other users when they run into problems.
Quick turnaround on fixes that can be designed, developed, reviewed and landed in less than a day.
An automation imperative - there's no way we could do this manually, so we are forced to build an infrastructure that can be deployed instantly and largely automatically. That results in better testing, better processes, better quality.
Trunk which is always ready to deploy. We know, on any given day, what the critical issues in trunk are, and if needed we can probably deploy the tip of trunk straight to production. It's very rare, but it has happened.
Passionate users are essential!
Probably the most important element of the beta program is that it is a team of nearly 1,000 people who have been interested enough in Launchpad to want to sign up to the team, receive some email and chat with others. We really should interact more directly with this team, it will be fantastic to have a mailing list for the team so folks who are *really* interested can sign up to that.
People do not stay passionate if they don't have active conversations with other passionate users. They want to feed this habit, they want to participate, they want to be drivers and conductors as much as passengers in this race. The beta program is a GREAT opportunity to make our best users feel like they are in the drivers seat, specifically with regard to features we HAVE ALREADY IMPLEMENTED. The great thing about the beta program is that the discussion is not about something we might schedule to start work on in six months, it's about something that is ABOUT to be revealed to the whole Launchpad audience. So ideas, tweaks, innovations, improvements are really timely.
We need to treat the beta team with some just-for-you surprises now and then. Every month, we want to have a special campaign of code that is on edge.launchpad.net but only visible to beta testers. Read http://headrush.typepad.com/creating_passionate_users/2005/05/the_case_for_ea.html if you haven't already - its all about giving your best users something special, something just for them.
Beta campaign best practices
So, down to the nuts and bolts. This is what it takes to run a really good special beta campaign. Every developer in Launchpad is entitled to have a feature they are working on get a special beta campaign. Consider it a privilege!
One feature at a time - we will only have a campaign for a single feature at a time. That means we have a laser-like focus with our beta team on a specific thing we are landing. It means that the email which goes out to them can focus on one and only one thing we want to discuss and test. A campaign can be as short as a week or as long as a full release cycle - we can run multiple campaigns in a single cycle if we are well prepared.
Personalized invitations to test the feature are sent to the beta testers when we start a special beta campaign. The mail can be sent by the lead developer, or someone else if that developer does not want to do so. The requirement is that whoever sends the mail replies to *every* response they get - from experience that's 20-50 emails.
The code is testable immediately by beta testers. The feature should be right there on edge, your email to the beta team members will point them to a few examples of people who have used it already (i.e. you
and it will also include the link where they can try it out immediately themselves. DO NOT run a beta program off staging, or off dogfood, or off demo, or any other special program. By all means, get Canonical folks to test your code in those places, but don't announce that or consider it a beta program. It is not a beta program if it is not in production and instantly accessible. Exclusivity to beta testers. The beta team is special - they get something that nobody else has, which is access to exclusive features before widespread release. If we make the features generally available, we take all the fun out of the beta program (and we risk having a feature flooded before it is ready). So even on edge, the menus, forms and pages that expose the feature should be conditionally filtered off. There is infrastructure in LP to make this easy.
Implementing beta campaigns
It's easy to run a beta campaign!
Land your database changes early, in the cycle before you plan to run the campaign. This gives you freedom and flexibility to start the beta program when you want. Your database changes can even be quite intrusive, just preserve the old behaviour. For example, loc.diff is a change that removed Person.timezone, moving it to a new table, while preserving the old read and write semantics of Person.timezone while the new functionality could be developed.
Discuss the beta campaign with team leads and project leads - we want to make sure your beta campaign gets the best reception and fits in with the schedule of other announcements going on at Canonical. We'll agree how long the campaign is likely to be, and whether there are other things we can do to draw attention to the feature.
Write your code and tests as normal - you don't have to do anything special. Try to isolate to some extent the visual changes that go along with your feature. New pages, or new portlets on pages, are all fine.
Make access to the features, menus, portlets, forms and pages conditional on beta membership. You can easily do this. Every LaunchpadView subclass has a view/isBetaUser attribute, so you can easily say <tal:is_beta condition="view/isBetaUser> or, in a menu, enabled=self.isBetaUser. In forms, you should also add a form-wide validation that verifies the same, so that people don't POST forms directly.
Your tests will need to have users that are in the beta team. Don't change anything, just put the users you have used for your tests into the beta team at the start of the tests. You will delete those lines in the landing which removes the "beta team only" restrictions. There used to be some problems with membership in the beta test team conferring additional permissions on the Ubuntu distro in the sample data, but that's no longer the case. You should test that the user cannot see the feature, then put the user in the beta team (in the test), then verify that they can see the feature and proceed. When you "pull the curtain away" you just delete the first test and the addition of the user to the beta team, nothing else in the test needs to change.
When ready, make an announcement. You can use the spam.py script to send your announcement to the beta testers. You'll need to get a LP admin to run a SQL query against the staging database. You can find everything you need in this branch: bzr+ssh://devpad.canonical.com/code/barry/massmail
Good luck, enjoy it!
