'''Note: this is an unofficial/wip LEP.'''

= Improve our Launchpad setup scripts =

Replace ''rocketfuel-*'' to produce a new script more easily maintainable, more
informative for the user, better tested, and more flexible--specifically, flexible enough to support both LXC and non-LXC development environments.

''lpsetup'' is a Python project currently used by the Yellow Squad to create a Launchpad testing 
environment inside a container, that is then used as a template for ephemeral 
instances. Starting isolated ephemeral instances, each one containing a full 
Launchpad environment, is actually the preferred way to [[https://dev.launchpad.net/LEP/ParallelTesting|run tests in parallel]]. 
''lpsetup'' can be improved in order to let developers create and update a 
full Launchpad environment, inside an LXC or just in the host. This way ''lpsetup'' 
can be considered a replacement of ''rocketfuel-*'' scripts.

'''Contact:''' [[https://launchpad.net/~frankban|frankban]]<<BR>>
'''On Launchpad:''' [[lpsetup|https://launchpad.net/lpsetup]]

== Rationale ==

Launchpad has a lot of dependencies, and the process of setting up and building 
the code can be difficult and painful.
The ''rocketfuel-*'' scripts do not have automated tests, take over the developer's system in many ways, have minimal interaction with the user as they are run, and are not written in Python. 

However, the effort the yellow squad made to code an automated Python script for parallel 
tests can be reused, with small improvements, to help developers run, fix and develop [[https://launchpad.net/launchpad|Launchpad itself]]. It can provide a tool to get started with 
Launchpad with much less pain, and in a way that leaves the host 
relatively untouched, because almost all of the dependencies can be installed inside a Linux container (LXC) if desired.

We hope that this will help community contributors begin Launchpad development more easily.  We hope that this will also give core developers an improved development experience, as well as a more maintainable set of tools.

As written above, we are doing this now because it seems to us a natural 
consequence of what we produced in the parallel testing project. We started 
creating the testing environment with ''setuplxc'', a standalone script first 
developed for parallel tests. As the time passed, it became evident that a 
re-factor was needed to let the process become more reliable and reusable. 
This re-factor was completed as a slack time project, and now ''lpsetup'' has 
fully replaced the deprecated (and, actually, deleted) ''setuplxc''.

== Stakeholders ==

All Launchpad developers.<<BR>>
The Launchpad contributors.

'''As a '''Developer<<BR>>
'''I want '''to easily create and update Launchpad development environments 
inside LXC containers<<BR>>
'''so that '''I can concentrate on fixing bugs and adding features without 
spending time and resources on the environment set up and configuration.<<BR>>

'''As a '''Developer<<BR>>
'''I want '''to have a full test suite for the code used to create and update 
Launchpad instances<<BR>>
'''so that '''when I change it, I can be confident that it still works.<<BR>>

'''As a '''Developer or Contributor<<BR>>
'''I want '''to be informed about all the changes needed in my system to set up
Launchpad instances<<BR>>
'''so that '''I can easily revert them, and acquire knowledge about the environment
and the technologies involved.<<BR>>

'''As a '''Contributor<<BR>>
'''I want '''a tool that lets me dig into Launchpad code and run the application<<BR>>
'''so that '''I can quickly start helping with development and contributing, 
without even dirty my base system.<<BR>>


== Constraints and Requirements ==

=== Must ===

 * A new contributor can make a new environment to work on Launchpad.  We have four scenarios, each of which lists associated steps that roughly represent some shell command.  All LXC scenarios intend to use a bind mounted home directory from the host.  If a command does not explicitly mention sudo, it must be runnable without sudo privileges.
   * Scenario: LXC, lightweight checkouts:
     * create/init lxc container (sudo)
     * ssh into container
     * get code (checkout), specifying location for creating bzr repo and specifying location for creating branch
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container" command)
   * Scenario: LXC, branches (including colocated) (Nice to have for Robert, but mostly already implemented)
     * create/init lxc container (sudo)
     * ssh into container
     * get code (branch), specifying location for creating bzr repo and specifying to make a branch (lightweight checkouts are default)
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container" command)
   * Scenario: local, lightweight checkouts:
     * init local environment (sudo)
     * get code (checkout), specifying location for creating bzr repo and specifying location for creating branch
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container"
   * Scenario: local, branches (including colocated) (Nice to have for Robert, but mostly already implemented)
     * init local environment (sudo)
     * get code (branch), specifying location for creating bzr repo and specifying to make a branch (lightweight checkouts are default)
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container"
 * A contributor can update his host container (sudo).
   * Scenario: LXC, checkouts or branches
     * run sudo update command in host (why? some cleanup may be necessary in LXC configuration.  It would also ssh into container and update within container)
   * Scenario: Local, checkouts or branches
     * run sudo update command in environment
 * A contributor can update his branches (not sudo)
   * Scenario: all
     * run a single command that wants to be run at top of LP tree.  Updates everything except sudo-bits like apt dependencies (this must be done when updating container).  Maybe it can warn if an update is necessary.
 * (Note that removing a container and entering a container can be done with lxc-destroy and ssh, respectively, and we do not intend to have custom commands for them at this time.)
 * keep lpsetup and lp-dev-utils separated. Reasons:
   * lp-dev-utils is more for lxc container and lpsetup is more for host.
   * Moving lpsetup to lp-dev-utils is a non-trivial task because it means packaging lp-dev-utils for lucid and precise.
   * Because we don't believe it is the right thing and this is a slack time task, we don't intend to do it.
 * Copy `launchpad-database-setup` to lpsetup and have `lp-setup init` run it (note that we will delete it from the Launchpad tree once lpsetup is released and blessed).
 * Warn about all the changes made by the program to the system (especially the host, in the LXC story), giving the user the chance to quit the process.
 * Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
 * Add banners to all created and modified files in the system.
 
=== Nice to have ===

 * A contributor can make a new branch in an existing branch-based repository & environment (scenario is LXC or host)
   * get code (branch).  No options are required to specify the repository (should be existing, in a parent directory), or that  a full branch rather than a lightweight checkout is desired (this should be able to be determined by identifying the repository as having not been created with --no-trees).
     * Optionally, you can specify another Launchpad branch (local or remote) other than the remote devel branch to use as a base.
     * Optionally, you can specify another local branch to use to link external sourcecode/download-cache.  Otherwise it will get everything afresh.
 * A contributor can make a new lightweight checkout in an existing checkout-based repository & environment (scenario is LXC or host)
   * get code (checkout).  You must (should? default to remote?) include an option to specify what local no-tree branch to use.  No options are required to specify the repository (should be existing, in a parent directory), or that a full branch rather than a lightweight checkout is desired (this should be able to be determined by identifying the repository as having not been created with --no-trees).
     * Optionally, you can specify another local branch to use to link external sourcecode/download-cache.  Otherwise it will get everything afresh.
 * A full integration test (e.g. using juju) supporting different options and use cases.
 * Separate commands (easy): create entry points so that a subcommand can be invoked as a script. This way we can have a central hub (lp-setup) and several other executables (lp-init, lp-update...), maybe created by the distribution scripts (setup.py).
 * Scripts should be idempotent.
 * Apache configuration and host configuration is moved to `lp-setup init` (currently this is done in `make install`).
 * Unlikely but cool: get host hosts file working correctly in lxc story; otherwise warn users that they must manually change /etc/hosts every time they start a container; or we can provide an lxc-start wrapper that starts lxc and changes /etc/hosts to match the current lxc ip address.
 * `lp-setup get` can be passed a directory to find the download cache and sourcecode.  If not supplied, a new version is obtained.
 * `lp-setup update` recognizes when sourcecode has symlinks and offers to do the right thing: update the source code where it really is and then link it in the current checkout.
 
=== Must not ===

 * Change the host environment without warning the user and without giving the user a chance to opt-out.
 * Create or modify a system-level configuration file without specifying the creation/modification source and date-time.

=== Out of scope ===
 
==== Hopefully soon after release ====

 * Remove launchpad-database-setup from Launchpad.
 * Package lp-dev-utils and have it depend on lpsetup.
 
==== Maybe sometime later ====

 * Add ''ec2'' lp-setup "container" type?
 * Add ''lxc-ephemeral'' lp-setup "container" type? 
 * Combine lp-dev-utils with lpsetup?
 * Replace other scripts under `utilities/` other than ''rocketfuel-*''.

==== Maybe never ====

 * Create Launchpad environments in systems != current Ubuntu release or current production release (the LTS on which Launchpad runs in the data center).

== Success ==

=== How will we know when we are done? ===

Developers can successfully use the project to create their development environment.<<BR>>
''lpsetup'' is accepted as part of the Launchpad project, ready to be maintained by the Launchpad team.<<BR>>
The removal of ''rocketfuel-*'' from the tree is accepted by the Launchpad team.
lpsetup can continue to be used as part of the parallel test process.

=== How will we measure how well we have done? ===

One month without critical bugs after the first release.<<BR>>
Feedback from developers and contributors.<<BR>>

== Notes ==

Design and implementation notes can be found [[https://dev.launchpad.net/yellow/lp-setup-implementation-notes|here]].

== Thoughts? ==

This LEP has been modified in response to comments and thoughts previously received: they can be found [[https://dev.launchpad.net/yellow/lp-setup-thoughts|here]].