Launchpad setup scripts: incorporated thoughts

See the LEP.

RobertCollins

tl;dr: I suggest limiting the scope to 'Machine or container setup for doing LP development, explicitly excluding 'obtain a copy of the source code'.'

I would rewrite the MUST's as:

• Be able to make system-wide changes needed to do development without containers.
• Be able to create an LXC container configured appropriate to permit Launchpad development to be done within it, assuming a bind-mounted ~.
• Warn about all the system-wide changes made by the program, giving the user the chance to quit the process. (e.g. 'this will create a container', 'this will install the following packages'). Modifications made in a new container do not need to be notified.
• Continue supporting our parallel testing story, including options for including tweaks, helpers, generated scripts, and a non-interactive execution.
• Add banners to all created and modified configuration files in the system.

Note that:

• I add configuration to the last option because files from debs and database instances etc can't be bannered. I think configuration is your intent.
• I've dropped 'store the state in a file' - you may choose to do that, its not clear to me that its something we must have to have an improvement over rocketfuel scripts : My intent in suggesting this is to allow you more freedom in execution - I think a stateful config file in e.g. /etc/lpsetup would be fine. [Some thought needed about what goes on the host and what goes in the container, when dealing with container setups]
• I've dropped 'replace the rocketfuel-*' scripts, because they are overloaded messes, and trying to directly replace them will just drive us into an overloaded mess: if we have a clean tool that can be driven directly, we can invoke it from rocketfuel-* to the extent that it does not replace the scripts, and this project stays focused.

Reasoning

The LEP seems to describe three intents:

1. set a machine up to develop LP
2. Update a development environment
3. Create branches

The first is complex and requires installing additional packages and changing the config of apache and postgresql (and more in future, unless/until we fixturise everything).

The second is pretty tightly related to the first, as knowing that e.g. the apache setup needs redoing is hard otherwise.

However the third item seems entirely unrelated : I suggest splitting the problem in two parts:

1. Making a correct setup for the source trees (lp:launchpad, ./sourcecode/*, download-cache/)
2. Making a correct machine setup for for the development/test environment (e.g. apache *does not* need to be altered for test, only for development).

This may imply two tools, or two submodules or something. Crucially though, your statement about putting non rocketfuel-* scripts out of scope, clearly rules out updating source trees (because update-sourcecode isn't a rocketfuel-* script... but its called from one - see below). It also makes it easier to reason about what happens inside a container (machine setup) and what happens whereever the user wants (source tree setup and maintenance). There are two connections between these components AIUI:

• The combo loader (may) point into the source tree [needs checking]
• The Database sample setup needs the source tree around to run

The former would only need the intended path for the source tree to be done, and the latter is something that 'make schema' already knows how to do - its not part of machine setup. So drawing a hard line around 'machine setup' seems like a better way to limit the scope than referring to what rocketfuel-* scripts do.

Notes from Gary and Francesco in preparation for a call with Robert about his reply

(These are cryptic, sorry; they are primarily intended to be mnemonic.)

1. Parallel testing needs to set up a repo and get a branch
2. A lot of the system setup is contained within the tree (e.g. apache setup is done from the Makefile). We like the idea of what is described but it will be more work, not less, in our estimation, to divorce these steps from the tree.
3. Agree with third note: "I've dropped 'replace the rocketfuel-*' scripts, because they are overloaded messes, and trying to directly replace them will just drive us into an overloaded mess: if we have a clean tool that can be driven directly, we can invoke it from rocketfuel-* to the extent that it does not replace the scripts, and this project stays focused." We want a command that sets up an environment, which includes a primed repository and an initial checkout that this process needs; we want a command that can update the initial environment; and we want a command that sets up branches for work. People can use the first while ignoring the second, and use the first two while ignoring the third--each depends on the previous one, but does not demand the subsequent one. Maybe Robert is arguing against third? Francesco and Gary think it would be good to have, but don't feel too strongly about it.
4. Agree with all of his MUSTs.
5. "store the state in a file": we appreciate the freedom, but we think we may have under-explained the idea. The idea is to make it possible to remember the options that are used at install time so that the branch and update commands can use them without passing all the options again. lpsetup already has a set of defaults now, though they are in the code atm (settings.py).
6. We agree that having a strong, clear line between a testing environment and a dev environment would be good. We already have that to some degree but it can be improved, both in the implementation (e.g. not installing Apache hooks for test instance) and possibly in the UI (maybe a separate command rather than an option?)

7. We are not quite sure what you mean by the last three or four paragraphs but we think we might have an idea, and we think we might even mostly agree. We'll talk about it on the call.