Note: this is an unofficial/wip LEP.

Improve our Launchpad setup scripts

Replace rocketfuel-* to produce a new script more easily maintainable, more informative for the user, better tested, and more flexible--specifically, flexible enough to support both LXC and non-LXC development environments.

lpsetup is a Python project currently used by the Yellow Squad to create a Launchpad testing environment inside a container, that is then used as a template for ephemeral instances. Starting isolated ephemeral instances, each one containing a full Launchpad environment, is actually the preferred way to run tests in parallel. lpsetup can be improved in order to let developers create and update a full Launchpad environment, inside an LXC or just in the host. This way lpsetup can be considered a replacement of rocketfuel-* scripts.

Contact: frankban
On Launchpad: https://launchpad.net/lpsetup

Rationale

Launchpad has a lot of dependencies, and the process of setting up and building the code can be difficult and painful. The rocketfuel-* scripts do not have automated tests, take over the developer's system in many ways, have minimal interaction with the user as they are run, and are not written in Python.

However, the effort the yellow squad made to code an automated Python script for parallel tests can be reused, with small improvements, to help developers run, fix and develop Launchpad itself. It can provide a tool to get started with Launchpad with much less pain, and in a way that leaves the host relatively untouched, because almost all of the dependencies can be installed inside a Linux container (LXC) if desired.

We hope that this will help community contributors begin Launchpad development more easily. We hope that this will also give core developers an improved development experience, as well as a more maintainable set of tools.

As written above, we are doing this now because it seems to us a natural consequence of what we produced in the parallel testing project. We started creating the testing environment with setuplxc, a standalone script first developed for parallel tests. As the time passed, it became evident that a re-factor was needed to let the process become more reliable and reusable. This re-factor was completed as a slack time project, and now lpsetup has fully replaced the deprecated (and, actually, deleted) setuplxc.

Stakeholders

All Launchpad developers.
The Launchpad contributors.

As a Developer
I want to easily create and update Launchpad development environments inside LXC containers
so that I can concentrate on fixing bugs and adding features without spending time and resources on the environment set up and configuration.

As a Developer
I want to have a full test suite for the code used to create and update Launchpad instances
so that when I change it, I can be confident that it still works.

As a Developer or Contributor
I want to be informed about all the changes needed in my system to set up Launchpad instances
so that I can easily revert them, and acquire knowledge about the environment and the technologies involved.

As a Contributor
I want a tool that lets me dig into Launchpad code and run the application
so that I can quickly start helping with development and contributing, without even dirty my base system.

Constraints and Requirements

Must

Nice to have

Must not

Out of scope

Hopefully soon after release

Maybe sometime later

Maybe never

Success

How will we know when we are done?

Developers can successfully use the project to create their development environment.
lpsetup is accepted as part of the Launchpad project, ready to be maintained by the Launchpad team.
The removal of rocketfuel-* from the tree is accepted by the Launchpad team. lpsetup can continue to be used as part of the parallel test process.

How will we measure how well we have done?

One month without critical bugs after the first release.
Feedback from developers and contributors.

Thoughts?

Make a separate page: Design and implementation notes

Proposed UI

The application will support 3 main sub commands:

  1. initialize a container (e.g. local, lxc, ec2, lxc-ephemeral)
  2. get the sourcecode and source dependencies
  3. update the environment

Trying to reuse as much of the current code as possible, we could create a system like the following:

  1. lp-setup init [container]

    • e.g.: lp-setup init lxc
    • Accepts a container argument: we could create a container object as an interface for really different containers to be created, started, stopped. The contract could also let the user connect and execute commands inside the container.
    • The container is initialized installing all the launchpad dependencies and the required tweaks, but no source code is retrieved.

  2. lp-setup get

    • Gets a full tree (LP, download-cache, source).
    • Supports lightweight checkouts (default) or --with-trees ("normal"/co-located branches).

  3. lp-setup update [branch]

    • Updates the current branch (or the given one): shelve, pull (or merge if necessary), unshelve if successful.
    • Also updates apt dependencies if requested.
    • Accepts a directory for the sourcecode as an option.
    • Updates and links external sourcecode.
    • Runs make/make schema if requested and if code changes are found.

To continue supporting our current parallel testing story, we also need a sub command that non-interactively creates and builds the environment, including source code. The name of this sub command must be decided (lp-setup testing?).

Implementation notes

The UI/entry-point of lpsetup is the lp-setup command. This command is made of several sub commands, and each sub command can contain different steps. When executing a sub command it is possible to specify the steps to run or to skip. This machinery is implemented using:

Make a separate page: Incorporated thoughts

RobertCollins

tl;dr: I suggest limiting the scope to 'Machine or container setup for doing LP development, explicitly excluding 'obtain a copy of the source code'.'

I would rewrite the MUST's as:

Note that:

Reasoning

The LEP seems to describe three intents:

  1. set a machine up to develop LP
  2. Update a development environment
  3. Create branches

The first is complex and requires installing additional packages and changing the config of apache and postgresql (and more in future, unless/until we fixturise everything).

The second is pretty tightly related to the first, as knowing that e.g. the apache setup needs redoing is hard otherwise.

However the third item seems entirely unrelated : I suggest splitting the problem in two parts:

  1. Making a correct setup for the source trees (lp:launchpad, ./sourcecode/*, download-cache/)
  2. Making a correct machine setup for for the development/test environment (e.g. apache *does not* need to be altered for test, only for development).

This may imply two tools, or two submodules or something. Crucially though, your statement about putting non rocketfuel-* scripts out of scope, clearly rules out updating source trees (because update-sourcecode isn't a rocketfuel-* script... but its called from one - see below). It also makes it easier to reason about what happens inside a container (machine setup) and what happens whereever the user wants (source tree setup and maintenance). There are two connections between these components AIUI:

The former would only need the intended path for the source tree to be done, and the latter is something that 'make schema' already knows how to do - its not part of machine setup. So drawing a hard line around 'machine setup' seems like a better way to limit the scope than referring to what rocketfuel-* scripts do.

Notes from Gary and Francesco in preparation for a call with Robert about his reply

(These are cryptic, sorry; they are primarily intended to be mnemonic.)

  1. Parallel testing needs to set up a repo and get a branch
  2. A lot of the system setup is contained within the tree (e.g. apache setup is done from the Makefile). We like the idea of what is described but it will be more work, not less, in our estimation, to divorce these steps from the tree.
  3. Agree with third note: "I've dropped 'replace the rocketfuel-*' scripts, because they are overloaded messes, and trying to directly replace them will just drive us into an overloaded mess: if we have a clean tool that can be driven directly, we can invoke it from rocketfuel-* to the extent that it does not replace the scripts, and this project stays focused." We want a command that sets up an environment, which includes a primed repository and an initial checkout that this process needs; we want a command that can update the initial environment; and we want a command that sets up branches for work. People can use the first while ignoring the second, and use the first two while ignoring the third--each depends on the previous one, but does not demand the subsequent one. Maybe Robert is arguing against third? Francesco and Gary think it would be good to have, but don't feel too strongly about it.
  4. Agree with all of his MUSTs.
  5. "store the state in a file": we appreciate the freedom, but we think we may have under-explained the idea. The idea is to make it possible to remember the options that are used at install time so that the branch and update commands can use them without passing all the options again. lpsetup already has a set of defaults now, though they are in the code atm (settings.py).
  6. We agree that having a strong, clear line between a testing environment and a dev environment would be good. We already have that to some degree but it can be improved, both in the implementation (e.g. not installing Apache hooks for test instance) and possibly in the UI (maybe a separate command rather than an option?)

  7. We are not quite sure what you mean by the last three or four paragraphs but we think we might have an idea, and we think we might even mostly agree. :-) We'll talk about it on the call.

yellow/lp-setup-lep-draft (last edited 2012-05-30 16:17:55 by gary)