Note: this is an unofficial/wip LEP.

Improve our Launchpad setup scripts

Replace rocketfuel-* to produce a new script more easily maintainable, more informative for the user, better tested, and more flexible--specifically, flexible enough to support both LXC and non-LXC development environments.

lpsetup is a Python project currently used by the Yellow Squad to create a Launchpad testing environment inside a container, that is then used as a template for ephemeral instances. Starting isolated ephemeral instances, each one containing a full Launchpad environment, is actually the preferred way to run tests in parallel. lpsetup can be improved in order to let developers create and update a full Launchpad environment, inside an LXC or just in the host. This way lpsetup can be considered a replacement of rocketfuel-* scripts.

Contact: frankban
On Launchpad: https://launchpad.net/lpsetup

Rationale

Launchpad has a lot of dependencies, and the process of setting up and building the code can be difficult and painful. The rocketfuel-* scripts do not have automated tests, take over the developer's system in many ways, have minimal interaction with the user as they are run, and are not written in Python.

However, the effort the yellow squad made to code an automated Python script for parallel tests can be reused, with small improvements, to help developers run, fix and develop Launchpad itself. It can provide a tool to get started with Launchpad with much less pain, and in a way that leaves the host relatively untouched, because almost all of the dependencies can be installed inside a Linux container (LXC) if desired.

We hope that this will help community contributors begin Launchpad development more easily. We hope that this will also give core developers an improved development experience, as well as a more maintainable set of tools.

As written above, we are doing this now because it seems to us a natural consequence of what we produced in the parallel testing project. We started creating the testing environment with setuplxc, a standalone script first developed for parallel tests. As the time passed, it became evident that a re-factor was needed to let the process become more reliable and reusable. This re-factor was completed as a slack time project, and now lpsetup has fully replaced the deprecated (and, actually, deleted) setuplxc.

Stakeholders

All Launchpad developers.
The Launchpad contributors.

As a Developer
I want to easily create and update Launchpad development environments inside LXC containers
so that I can concentrate on fixing bugs and adding features without spending time and resources on the environment set up and configuration.

As a Developer
I want to have a full test suite for the code used to create and update Launchpad instances
so that when I change it, I can be confident that it still works.

As a Developer or Contributor
I want to be informed about all the changes needed in my system to set up Launchpad instances
so that I can easily revert them, and acquire knowledge about the environment and the technologies involved.

As a Contributor
I want a tool that lets me dig into Launchpad code and run the application
so that I can quickly start helping with development and contributing, without even dirty my base system.

Constraints and Requirements

Must

A new contributor can make a new environment to work on Launchpad. We have four scenarios, each of which lists associated steps that roughly represent some shell command. All LXC scenarios intend to use a bind mounted home directory from the host. If a command does not explicitly mention sudo, it must be runnable without sudo privileges.
- Scenario: LXC, lightweight checkouts:
  - create/init lxc container (sudo)
  - ssh into container
  - get code (checkout), specifying location for creating bzr repo and specifying location for creating branch
  - make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container" command)
- Scenario: LXC, branches (including colocated) (Nice to have for Robert, but mostly already implemented)
  - create/init lxc container (sudo)
  - ssh into container
  - get code (branch), specifying location for creating bzr repo and specifying to make a branch (lightweight checkouts are default)
  - make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container" command)
- Scenario: local, lightweight checkouts:
  - init local environment (sudo)
  - get code (checkout), specifying location for creating bzr repo and specifying location for creating branch
  - make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container"
- Scenario: local, branches (including colocated) (Nice to have for Robert, but mostly already implemented)
  - init local environment (sudo)
  - get code (branch), specifying location for creating bzr repo and specifying to make a branch (lightweight checkouts are default)
  - make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container"
A contributor can update his host container (sudo).
- Scenario: LXC, checkouts or branches
  - run sudo update command in host (why? some cleanup may be necessary in LXC configuration. It would also ssh into container and update within container)
- Scenario: Local, checkouts or branches
  - run sudo update command in environment
A contributor can update his branches (not sudo)
- Scenario: all
  - run a single command that wants to be run at top of LP tree. Updates everything except sudo-bits like apt dependencies (this must be done when updating container). Maybe it can warn if an update is necessary.
(Note that removing a container and entering a container can be done with lxc-destroy and ssh, respectively, and we do not intend to have custom commands for them at this time.)
keep lpsetup and lp-dev-utils separated. Reasons:
- lp-dev-utils is more for lxc container and lpsetup is more for host.
- Moving lpsetup to lp-dev-utils is a non-trivial task because it means packaging lp-dev-utils for lucid and precise.
- Because we don't believe it is the right thing and this is a slack time task, we don't intend to do it.
Copy launchpad-database-setup to lpsetup and have lp-setup init run it (note that we will delete it from the Launchpad tree once lpsetup is released and blessed).
Warn about all the changes made by the program to the system (especially the host, in the LXC story), giving the user the chance to quit the process.
Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
Add banners to all created and modified files in the system.

Nice to have

A contributor can make a new branch in an existing branch-based repository & environment (scenario is LXC or host)
- get code (branch). No options are required to specify the repository (should be existing, in a parent directory), or that a full branch rather than a lightweight checkout is desired (this should be able to be determined by identifying the repository as having not been created with --no-trees).
  - Optionally, you can specify another Launchpad branch (local or remote) other than the remote devel branch to use as a base.
  - Optionally, you can specify another local branch to use to link external sourcecode/download-cache. Otherwise it will get everything afresh.
A contributor can make a new lightweight checkout in an existing checkout-based repository & environment (scenario is LXC or host)
- get code (checkout). You must (should? default to remote?) include an option to specify what local no-tree branch to use. No options are required to specify the repository (should be existing, in a parent directory), or that a full branch rather than a lightweight checkout is desired (this should be able to be determined by identifying the repository as having not been created with --no-trees).
  - Optionally, you can specify another local branch to use to link external sourcecode/download-cache. Otherwise it will get everything afresh.
A full integration test (e.g. using juju) supporting different options and use cases.
Separate commands (easy): create entry points so that a subcommand can be invoked as a script. This way we can have a central hub (lp-setup) and several other executables (lp-init, lp-update...), maybe created by the distribution scripts (setup.py).
Scripts should be idempotent.
Apache configuration and host configuration is moved to lp-setup init (currently this is done in make install).
Unlikely but cool: get host hosts file working correctly in lxc story; otherwise warn users that they must manually change /etc/hosts every time they start a container; or we can provide an lxc-start wrapper that starts lxc and changes /etc/hosts to match the current lxc ip address.
lp-setup get can be passed a directory to find the download cache and sourcecode. If not supplied, a new version is obtained.
lp-setup update recognizes when sourcecode has symlinks and offers to do the right thing: update the source code where it really is and then link it in the current checkout.

Must not

Change the host environment without warning the user and without giving the user a chance to opt-out.
Create or modify a system-level configuration file without specifying the creation/modification source and date-time.

Out of scope

Hopefully soon after release

Remove launchpad-database-setup from Launchpad.
Package lp-dev-utils and have it depend on lpsetup.

Maybe sometime later

Add ec2 lp-setup "container" type?
Add lxc-ephemeral lp-setup "container" type?
Combine lp-dev-utils with lpsetup?
Replace other scripts under utilities/ other than rocketfuel-*.

Maybe never

Create Launchpad environments in systems != current Ubuntu release or current production release (the LTS on which Launchpad runs in the data center).

Success

How will we know when we are done?

Developers can successfully use the project to create their development environment.
lpsetup is accepted as part of the Launchpad project, ready to be maintained by the Launchpad team.
The removal of rocketfuel-* from the tree is accepted by the Launchpad team. lpsetup can continue to be used as part of the parallel test process.

How will we measure how well we have done?

One month without critical bugs after the first release.
Feedback from developers and contributors.

Notes

Design and implementation notes can be found here.

Thoughts?

This LEP has been modified in response to comments and thoughts previously received: they can be found here.

launchpad development

LEP/LaunchpadSetupScripts