Note: this is an unofficial/wip LEP.

Improve our Launchpad setup scripts

Replace rocketfuel-* to produce a new script more easily maintainable, more informative for the user, better tested, and more flexible--specifically, flexible enough to support both LXC and non-LXC development environments.

lpsetup is a Python project currently used by the Yellow Squad to create a Launchpad testing environment inside a container, that is then used as a template for ephemeral instances. Starting isolated ephemeral instances, each one containing a full Launchpad environment, is actually the preferred way to run tests in parallel. lpsetup can be improved in order to let developers create and update a full Launchpad environment, inside an LXC or just in the host. This way lpsetup can be considered a replacement of rocketfuel-* scripts.

Contact: frankban
On Launchpad: https://launchpad.net/lpsetup

Rationale

Launchpad has a lot of dependencies, and the process of setting up and building the code can be difficult and painful. The rocketfuel-* scripts do not have automated tests, take over the developer's system in many ways, have minimal interaction with the user as they are run, and are not written in Python.

However, the effort the yellow squad made to code an automated Python script for parallel tests can be reused, with small improvements, to help developers run, fix and develop Launchpad itself. It can provide a tool to get started with Launchpad with much less pain, and in a way that leaves the host relatively untouched, because almost all of the dependencies can be installed inside a Linux container (LXC) if desired.

We hope that this will help community contributors begin Launchpad development more easily. We hope that this will also give core developers an improved development experience, as well as a more maintainable set of tools.

As written above, we are doing this now because it seems to us a natural consequence of what we produced in the parallel testing project. We started creating the testing environment with setuplxc, a standalone script first developed for parallel tests. As the time passed, it became evident that a re-factor was needed to let the process become more reliable and reusable. This re-factor was completed as a slack time project, and now lpsetup has fully replaced the deprecated (and, actually, deleted) setuplxc.

Stakeholders

All Launchpad developers.
The Launchpad contributors.

As a Developer
I want to easily create and update Launchpad development environments inside LXC containers
so that I can concentrate on fixing bugs and adding features without spending time and resources on the environment set up and configuration.

As a Developer
I want to have a full test suite for the code used to create and update Launchpad instances
so that when I change it, I can be confident that it still works.

As a Developer or Contributor
I want to be informed about all the changes needed in my system to set up Launchpad instances
so that I can easily revert them, and acquire knowledge about the environment and the technologies involved.

As a Contributor
I want a tool that lets me dig into Launchpad code and run the application
so that I can quickly start helping with development and contributing, without even dirty my base system.

Constraints and Requirements

Must

• Optionally create Launchpad environments in the host machine.
• Optionally create Launchpad environments inside LXC containers.

• Replace rocketfuel-get: update the Launchpad environment.

• Replace rocketfuel-branch: create a new branch of devel.

• Warn about all the changes made by the program to the system, giving the user the chance to quit the process.
• Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
• Add banners to all created and modified files in the system.
• Store the initial configuration in a file, so that a saved state can be used when other commands are invoked.

Nice to have

• A way to store different configurations for different environments (e.g. adding a --remember option).
• A full integration test (e.g. using juju) supporting different options and use cases.
• Update and branch from the host, even if the container is not running.

• A bzr-like UI (i.e. lp-setup subcommand [options])

Must not

• Change the host environment without warning the user and without giving the user a chance to opt-out.
• Create or modify a file without specifying the creation/modification source and date-time.

Out of scope

• Replace scripts under utilities/ other than rocketfuel-*.

• Support all the customized environments that can possibly be used by Launchpad developers (i.e. lightweight checkouts).

• Create Launchpad environments in ec2 instances (we have the ec2 scripts, juju, etc.).

• Create Launchpad environments in systems != current Ubuntu release or current production release.

Success

How will we know when we are done?

Developers can successfully use the project to create their development environment.
lpsetup is accepted as part of the Launchpad project, ready to be maintained by the Launchpad team.
The removal of rocketfuel-* from the tree is accepted by the Launchpad team.

How will we measure how well we have done?

One month without critical bugs after the first release.
Feedback from developers and contributors.
When lpsetup is used as part of the parallel tests process, being able to reliably start test runs using buildbot is a good evidence that everything is set up well.

Thoughts?

RobertCollins

tl;dr: I suggest limiting the scope to 'Machine or container setup for doing LP development, explicitly excluding 'obtain a copy of the source code'.'

I would rewrite the MUST's as:

• Be able to make system-wide changes needed to do development without containers.
• Be able to create an LXC container configured appropriate to permit Launchpad development to be done within it, assuming a bind-mounted ~.
• Warn about all the system-wide changes made by the program, giving the user the chance to quit the process. (e.g. 'this will create a container', 'this will install the following packages'). Modifications made in a new container do not need to be notified.
• Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
• Add banners to all created and modified configuration files in the system.

Note that:

• I add configuration to the last option because files from debs and database instances etc can't be bannered. I think configuration is your intent.
• I've dropped 'store the state in a file' - you may choose to do that, its not clear to me that its something we must have to have an improvement over rocketfuel scripts : My intent in suggesting this is to allow you more freedom in execution - I think a stateful config file in e.g. /etc/lpsetup would be fine. [Some thought needed about what goes on the host and what goes in the container, when dealing with container setups]
• I've dropped 'replace the rocketfuel-*' scripts, because they are overloaded messes, and trying to directly replace them will just drive us into an overloaded mess: if we have a clean tool that can be driven directly, we can invoke it from rocketfuel-* to the extent that it does not replace the scripts, and this project stays focused.

Reasoning

The LEP seems to describe three intents:

1. set a machine up to develop LP
2. Update a development environment
3. Create branches

The first is complex and requires installing additional packages and changing the config of apache and postgresql (and more in future, unless/until we fixturise everything).

The second is pretty tightly related to the first, as knowing that e.g. the apache setup needs redoing is hard otherwise.

However the third item seems entirely unrelated : I suggest splitting the problem in two parts:

1. Making a correct setup for the source trees (lp:launchpad, ./sourcecode/*, download-cache/)
2. Making a correct machine setup for for the development/test environment (e.g. apache *does not* need to be altered for test, only for development).

This may imply two tools, or two submodules or something. Crucially though, your statement about putting non rocketfuel-* scripts out of scope, clearly rules out updating source trees (because update-sourcecode isn't a rocketfuel-* script... but its called from one - see below). It also makes it easier to reason about what happens inside a container (machine setup) and what happens whereever the user wants (source tree setup and maintenance). There are two connections between these components AIUI:

• The combo loader (may) point into the source tree [needs checking]
• The Database sample setup needs the source tree around to run

The former would only need the intended path for the source tree to be done, and the latter is something that 'make schema' already knows how to do - its not part of machine setup. So drawing a hard line around 'machine setup' seems like a better way to limit the scope than referring to what rocketfuel-* scripts do.