'''Note: this is an unofficial/wip LEP.'''

= Improve our Launchpad setup scripts =

Replace ''rocketfuel-*'' to produce a new script more easily maintainable, more
informative for the user, better tested, and more flexible--specifically, flexible enough to support both LXC and non-LXC development environments.

''lpsetup'' is a Python project currently used by the Yellow Squad to create a Launchpad testing 
environment inside a container, that is then used as a template for ephemeral 
instances. Starting isolated ephemeral instances, each one containing a full 
Launchpad environment, is actually the preferred way to [[https://dev.launchpad.net/LEP/ParallelTesting|run tests in parallel]]. 
''lpsetup'' can be improved in order to let developers create and update a 
full Launchpad environment, inside an LXC or just in the host. This way ''lpsetup'' 
can be considered a replacement of ''rocketfuel-*'' scripts.

'''Contact:''' [[https://launchpad.net/~frankban|frankban]]<<BR>>
'''On Launchpad:''' [[lpsetup|https://launchpad.net/lpsetup]]

== Rationale ==

Launchpad has a lot of dependencies, and the process of setting up and building 
the code can be difficult and painful.
The ''rocketfuel-*'' scripts do not have automated tests, take over the developer's system in many ways, have minimal interaction with the user as they are run, and are not written in Python. 

However, the effort the yellow squad made to code an automated Python script for parallel 
tests can be reused, with small improvements, to help developers run, fix and develop [[https://launchpad.net/launchpad|Launchpad itself]]. It can provide a tool to get started with 
Launchpad with much less pain, and in a way that leaves the host 
relatively untouched, because almost all of the dependencies can be installed inside a Linux container (LXC) if desired.

We hope that this will help community contributors begin Launchpad development more easily.  We hope that this will also give core developers an improved development experience, as well as a more maintainable set of tools.

As written above, we are doing this now because it seems to us a natural 
consequence of what we produced in the parallel testing project. We started 
creating the testing environment with ''setuplxc'', a standalone script first 
developed for parallel tests. As the time passed, it became evident that a 
re-factor was needed to let the process become more reliable and reusable. 
This re-factor was completed as a slack time project, and now ''lpsetup'' has 
fully replaced the deprecated (and, actually, deleted) ''setuplxc''.

== Stakeholders ==

All Launchpad developers.<<BR>>
The Launchpad contributors.

'''As a '''Developer<<BR>>
'''I want '''to easily create and update Launchpad development environments 
inside LXC containers<<BR>>
'''so that '''I can concentrate on fixing bugs and adding features without 
spending time and resources on the environment set up and configuration.<<BR>>

'''As a '''Developer<<BR>>
'''I want '''to have a full test suite for the code used to create and update 
Launchpad instances<<BR>>
'''so that '''when I change it, I can be confident that it still works.<<BR>>

'''As a '''Developer or Contributor<<BR>>
'''I want '''to be informed about all the changes needed in my system to set up
Launchpad instances<<BR>>
'''so that '''I can easily revert them, and acquire knowledge about the environment
and the technologies involved.<<BR>>

'''As a '''Contributor<<BR>>
'''I want '''a tool that lets me dig into Launchpad code and run the application<<BR>>
'''so that '''I can quickly start helping with development and contributing, 
without even dirty my base system.<<BR>>


== Constraints and Requirements ==

=== Must ===

 * A new contributor can have an environment to work on Launchpad.  We have four scenarios, each of which lists associated steps that roughly represent some shell command.  All LXC scenarios intend to use a bind mounted home directory from the host.  If a command does not explicitly mention sudo, it must be runnable without sudo privileges.
   * Scenario: LXC, lightweight checkouts:
     * create/init lxc container (sudo)
     * ssh into container
     * get code (checkout), specifying location for creating bzr repo and specifying location for creating branch
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container" command)
   * Scenario: LXC, branches (including colocated) (Nice to have for Robert, but mostly already implemented)
     * create/init lxc container (sudo)
     * ssh into container
     * get code (branch), specifying location for creating bzr repo and specifying to make a branch (lightweight checkouts are default)
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container" command)
   * Scenario: local, lightweight checkouts:
     * init local environment (sudo)
     * get code (checkout), specifying location for creating bzr repo and specifying location for creating branch
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container"
   * Scenario: local, branches (including colocated) (Nice to have for Robert, but mostly already implemented)
     * init local environment (sudo)
     * get code (branch), specifying location for creating bzr repo and specifying to make a branch (lightweight checkouts are default)
     * make (for now, sudo make install && make schema; eventually sudo make install should be folded into the "init container"
 * A contributor can update his host container (sudo).
   * Scenario: LXC, checkouts or branches
     * run sudo update command in host (why? some cleanup may be necessary in LXC configuration.  It would also ssh into container and update within container)
   * Scenario: Local, checkouts or branches
     * run sudo update command in environment
 * A contributor can update his branches (not sudo)
   * Scenario: all
     * run a single command that wants to be run at top of LP tree.  Updates everything except sudo-bits like apt dependencies (this must be done when updating container).  Maybe it can warn if an update is necessary.
 * (Note that removing a container and entering a container can be done with lxc-destroy and ssh, respectively, and we do not intend to have custom commands for them at this time.)
 * keep lpsetup and lp-dev-utils separated. Reasons:
   * lp-dev-utils is more for lxc container and lpsetup is more for host.
   * Moving lpsetup to lp-dev-utils is a non-trivial task because it means packaging lp-dev-utils for lucid and precise.
   * Because we don't believe it is the right thing and this is a slack time task, we don't intend to do it.
 * Copy `launchpad-database-setup` to lpsetup and have `lp-setup init` run it.
 * Warn about all the changes made by the program to the system, giving the user the chance to quit the process.
 * Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
 * Add banners to all created and modified files in the system.
 * Store the initial configuration in a file, so that a saved state can be used when other commands are invoked.
 
=== Nice to have ===

 * A full integration test (e.g. using juju) supporting different options and use cases.
 * Separate commands (easy): create entry points so that a subcommand can be invoked as a script. This way we can have a central hub (lp-setup) and several other executables (lp-init, lp-update...), maybe created by the distribution scripts (setup.py).
 * Scripts should be idempotent.
 * Apache configuration and host configuration is moved to `lp-setup init` (currently this is done in `make install`).
 * Unlikely but cool: get host hosts file working correctly in lxc story; otherwise warn users that they must manually change /etc/hosts every time they start a container; or we can provide an lxc-start wrapper that starts lxc and changes /etc/hosts to match the current lxc ip address.
 * `lp-setup get` can be passed a directory to find the download cache and sourcecode.  If not supplied, a new version is obtained.
 * `lp-setup update` recognizes when sourcecode has symlinks and offers to do the right thing: update the source code where it really is and then link it in the current checkout.
 
=== Must not ===

 * Change the host environment without warning the user and without giving the user a chance to opt-out.
 * Create or modify a system-level configuration file without specifying the creation/modification source and date-time.

=== Out of scope ===
 
==== Hopefully soon after release ====

 * Remove launchpad-database-setup from Launchpad.
 * Package lp-dev-utils and have it depend on lpsetup.
 
==== Maybe sometime later ====

 * Add ''ec2'' lp-setup "container" type?
 * Add ''lxc-ephemeral'' lp-setup "container" type? 
 * Combine lp-dev-utils with lpsetup?
 * Replace other scripts under `utilities/` other than ''rocketfuel-*''.

==== Maybe never ====

 * Create Launchpad environments in systems != current Ubuntu release or current production release (the LTS on which Launchpad runs in the data center).

== Success ==

=== How will we know when we are done? ===

Developers can successfully use the project to create their development environment.<<BR>>
''lpsetup'' is accepted as part of the Launchpad project, ready to be maintained by the Launchpad team.<<BR>>
The removal of ''rocketfuel-*'' from the tree is accepted by the Launchpad team.
lpsetup can continue to be used as part of the parallel test process.

=== How will we measure how well we have done? ===

One month without critical bugs after the first release.<<BR>>
Feedback from developers and contributors.<<BR>>

== Thoughts? ==

= Make a separate page: Design and implementation notes =



== Proposed UI ==
  
The application will support '''3 main sub commands''':

 1. initialize a container (e.g. local, lxc, ec2, lxc-ephemeral)
 2. get the sourcecode and source dependencies
 3. update the environment

Trying to reuse as much of the current code as possible, we could create
a system like the following:

 1. `lp-setup init [container]`

    * e.g.: lp-setup init lxc
    * Accepts a container argument: we could create a container object as
    an interface for really different containers to be created, started,
    stopped. The contract could also let the user connect and execute
    commands inside the container. 
    * The container is initialized installing
    all the launchpad dependencies and the required tweaks, but no source
    code is retrieved.

 2. `lp-setup get`

    * Gets a full tree (LP, download-cache, source).
    * Supports lightweight checkouts (default) or --with-trees ("normal"/co-located branches).

 3. `lp-setup update [branch]`

    * Updates the current branch (or the given one): shelve, pull (or merge if necessary), unshelve if successful. 
    * Also updates apt dependencies if requested. 
    * Accepts a directory for the sourcecode as an option.
    * Updates and links external sourcecode.
    * Runs make/make schema if requested and if code changes are found.

To continue supporting our current '''parallel testing story''', we also need a sub command that non-interactively creates and builds the environment, including source code. The name of this sub command must be decided (`lp-setup testing`?).

== Implementation notes ==

The UI/entry-point of lpsetup is the `lp-setup` command. This command is made of several sub commands, and each sub command can contain different steps. 
When executing a sub command it is possible to specify the steps to run or to skip.
This machinery is implemented using:

 * a custom ''ArgumentParser'' (argparse.ArgumentParser):
   * with the ability to register and handle sub commands
   * with the ability to re-create the cmd line args from a namespace object (this is useful when the script run `sudo` on itself)

 * a ''SubCommand'' object:
   * supports namespace initialization and validation
   * implements the ''restart myself as root'' functionality
   * contains sub command help
   * adds to the parser the required options
   * optionally implements the steps system

= Make a separate page: Incorporated thoughts =

=== RobertCollins ===

tl;dr: I suggest limiting the scope to 'Machine or container setup for doing LP development, explicitly excluding 'obtain a copy of the source code'.'

I would rewrite the MUST's as:
 * Be able to make system-wide changes needed to do development without containers.
 * Be able to create an LXC container configured appropriate to permit Launchpad development to be done within it, assuming a bind-mounted ~.
 * Warn about all the system-wide changes made by the program, giving the user the chance to quit the process. (e.g. 'this will create a container', 'this will install the following packages'). Modifications made in a new container do not need to be notified.
 * Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
 * Add banners to all created and modified configuration files in the system. 

Note that:
 * I add configuration to the last option because files from debs and database instances etc can't be bannered. I think configuration is your intent.
 * I've dropped 'store the state in a file' - you may choose to do that, its not clear to me that its something we must have to have an improvement over rocketfuel scripts : My intent in suggesting this is to allow you more freedom in execution - I think a stateful config file in e.g. /etc/lpsetup would be fine. [Some thought needed about what goes on the host and what goes in the container, when dealing with container setups]
 * I've dropped 'replace the rocketfuel-*' scripts, because they are overloaded messes, and trying to directly replace them will just drive us into an overloaded mess: if we have a clean tool that can be driven directly, we can invoke it from rocketfuel-* to the extent that it does not replace the scripts, and this project stays focused.

==== Reasoning ====
The LEP seems to describe three intents:
 1. set a machine up to develop LP
 1. Update a development environment
 1. Create branches

The first is complex and requires installing additional packages and changing the config of apache and postgresql (and more in future, unless/until we fixturise everything). 

The second is pretty tightly related to the first, as knowing that e.g. the apache setup needs redoing is hard otherwise.

However the third item seems entirely unrelated : I suggest splitting the problem in two parts:
 1. Making a correct setup for the source trees (lp:launchpad, ./sourcecode/*, download-cache/)
 1. Making a correct machine setup for for the development/test environment (e.g. apache *does not* need to be altered for test, only for development).

This may imply two tools, or two submodules or something. Crucially though, your statement about putting non rocketfuel-* scripts out of scope, clearly rules out updating source trees (because update-sourcecode isn't a rocketfuel-* script... but its called from one - see below). It also makes it easier to reason about what happens inside a container (machine setup) and what happens whereever the user wants (source tree setup and maintenance). There are two connections between these components AIUI:
 * The combo loader (may) point into the source tree [needs checking]
 * The Database sample setup needs the source tree around to run

The former would only need the intended path for the source tree to be done, and the latter is something that 'make schema' already knows how to do - its not part of machine setup. So drawing a hard line around 'machine setup' seems like a better way to limit the scope than referring to what rocketfuel-* scripts do.

== Notes from Gary and Francesco in preparation for a call with Robert about his reply ==

(These are cryptic, sorry; they are primarily intended to be mnemonic.)

 1. Parallel testing needs to set up a repo and get a branch
 1. A lot of the system setup is contained within the tree (e.g. apache setup is done from the Makefile).  We like the idea of what is described but it will be more work, not less, in our estimation, to divorce these steps from the tree.
 1. Agree with third note: "I've dropped 'replace the rocketfuel-*' scripts, because they are overloaded messes, and trying to directly replace them will just drive us into an overloaded mess: if we have a clean tool that can be driven directly, we can invoke it from rocketfuel-* to the extent that it does not replace the scripts, and this project stays focused."  We want a command that sets up an environment, which includes a primed repository and an initial checkout that this process needs; we want a command that can update the initial environment; and we want a command that sets up branches for work.  People can use the first while ignoring the second, and use the first two while ignoring the third--each depends on the previous one, but does not demand the subsequent one.  Maybe Robert is arguing against third?  Francesco and Gary think it would be good to have, but don't feel too strongly about it.
 1. Agree with all of his MUSTs.
 1. "store the state in a file": we appreciate the freedom, but we think we may have under-explained the idea.  The idea is to make it possible to remember the options that are used at install time so that the branch and update commands can use them without passing all the options again.  lpsetup already has a set of defaults now, though they are in the code atm (settings.py).
 1. We agree that having a strong, clear line between a testing environment and a dev environment would be good. We already have that to some degree but it can be improved, both in the implementation (e.g. not installing Apache hooks for test instance) and possibly in the UI (maybe a separate command rather than an option?)
 1. We are not quite sure what you mean by the last three or four paragraphs but we think we might have an idea, and we think we might even mostly agree. :-) We'll talk about it on the call.