Launchpad Juju Charm for Developers

Make a Juju charm that can support casual/quick Launchpad development and one-off parallel testing (as opposed to automatic parallel testing).

Build with LEP/LaunchpadSetupScripts and in support of LEP/ParallelEC2Command.

Launchpad deployment with Juju is out of scope.

Contact: Gary Poster
On Launchpad: XXX Link to a blueprint, milestone or (best) a bug tag search across launchpad-project

Rationale

We needed something like this when Graham gave a UDS training session to teach and entice potential Launchpad contributors who have small itches to scratch. Now, when dedicated Launchpad development has reduced dramatically, enabling easier self-help for Launchpad's customers is a compelling goal to ease their pain and add to Launchpad's velocity.

It might also be useful for teams officially assigned to work on Launchpad: they could easily and quickly have another development environment available. That might be useful alone, to try and duplicate and investigate an issue; and it also might be useful for them to have a more powerful machine available, such as the 32-thread EC2 machines.

This also is a step along the way toward the proposed approach of LEP/ParallelEC2Command.

Stakeholders

The Launchpad developers are the primary stakeholders for this LEP. The LEP was developed after discussion with Graham Binns, who could have used something like this for the 12.10 UDS session, and after discussion with other Launchpad developers. The LEP will be announced to the launchpad-dev list for comments.

Another group of stakeholders is Launchpad users. Graham Binns has been used as a proxy for this group, but if this is regarded as insufficient then we can call for more comments.

User stories

Launchpad User Learns How to Develop Launchpad

As a Launchpad User
I want to quickly get a temporary development environment
so that I can be taught how to develop Launchpad at a live event such as a UDS session

Launchpad User Develops and Contributes a Quick Patch

As a Launchpad User
I want to quickly get a temporary development environment
so that I can develop, test, and push a Launchpad branch to quickly scratch an itch I have

Launchpad Developer Wants Another Machine to Investigate an Issue

As a Launchpad Developer
I want to quickly get a temporary development environment
so that I can investigate an issue, perhaps to see if I can duplicate a problem that I see locally within a clean environment.

Launchpad Developer Wants a More Powerful Machine for Tests

As a Launchpad Developer
I want to quickly get a temporary development environment
so that I can interactively run tests more quickly, probably in parallel but maybe just a subset of the full suite.

Constraints and Requirements

Must

• Provide a working development and testing environment in under five minutes

• Configure the environment to be able to use parallel testing (that is, the developer should be able to run tests with testr --parallel)

• Provide a mechanism to update any persistent data (such as an EBS snapshot) that the charm uses
• Work in EC2
• Be extendable to other environments, such as Canonistack, in the future.

Nice to have

• Provide support initially for Canonistack
• Provide support initially for alternate environments such as HP and Azure
• Have the Launchpad charms and any associated subordinate charms published in the official charm repository.
• Support juju expose to let people see a running Launchpad environment. This might be a little tricky because Launchpad will be running in an LXC, but I think this is a solved problem.

Must not

• Require people to copy over private SSH keys to the cloud in order to branch code from Launchpad or push code to Launchpad. shh -A should be sufficient.

Out of scope

• Any thoughts, preparation or discussion having to do with making this charm suitable for production deployment. Rationale: preparing for production deployment is a much more difficult task, and one that has many more stakeholders that will introduce many requirements unnecessary for these goals (e.g. Puppet integration).

Subfeatures

We intend to use LEP/LaunchpadSetupScripts for the initialization.

Success

How will we know when we are done?

A developer can enter "juju bootstrap" and a "juju deploy..." command or two and within five minutes have a Launchpad development environment in which she can quickly and successfully branch from and push to Launchpad. It would be OK if the developer also had to check out the charm and run it locally, rather than having it available in the official Juju repository, though that would be nice. The user may also have to use "juju ssh -A ..." to get to the machine, and may have to follow a short set of instructions, such as running "bzr launchpad-login" on the remote machine.

LEP/ParallelEC2Command can use this charm to automatically run tests.

How will we measure how well we have done?

Developers and users report success doing the above.

Thoughts?

Implementation/research Notes

Observations:

• Setting up a full Launchpad development environment from scratch takes about an hour. This is too long for our goals.
• One approach to making this go faster is to use AMIs. Juju does not support that, and is unlikely to AIUI because doing so would make Juju less platform-agnostic.
• Another approach to making this go faster is to use EBS snapshots. There are probably many ways to get this to hang together, but the following notes describe the approach that we investigated before actually proposing this LEP.

Current Idea:

• Launchpad developer charm has a required relationship with a LXC storage provider subordinate charm.
• Launchpad charm does not start to build/update Launchpad developer environment until storage provider subordinate charm joins.
• Launchpad charm uses lpsetup to update or build environment. If it finds an lxc environment (see the "fast" story below), it updates it; otherwise (see the "slow" story) it builds one.
• The "fast" deploy story is supplied by a subordinate charm that mounts a storage based on an existing EBS snapshot. The storage is then further bind mounted to provide both the home directory of the "ubuntu" user and /var/lib/lxc directory.
• The "update" or slow deploy story is supplied by a subordinate charm that mounts an empty storage and again bind mounts it to provide both the home directory of the "ubuntu" user and /var/lib/lxc directory. This is used to make a fresh snapshot to update the "fast" deploy story.
• Nice to have: the charm has an option to automatically create a new snapshot after the update or install is complete.
• Other LXC storage provider subordinate charms could be developed for other environments in the future, such as canonistack.

Verification:

Benji followed the above plan manually to verify that it would work, and figured out some of the details. The following are some of his notes on how to mount an EBS volume on an EC2 instance.

• Make the EBS volume and the EC2 instance (make sure they are in the same availability zone), associate the volume with the instance, and then do some mounting:

   sudo mke2fs -F -j /dev/xvdf
   sudo mkdir /mnt/ebs
   sudo mount /dev/xvdf /mnt/ebs

• Set up an EBS volume so LXC data will go there instead of the normal mount:

   sudo mkdir -p /mnt/ebs/home/ubuntu
   sudo cp -Rp . /mnt/ebs/home/ubuntu/
   sudo mount --bind /mnt/ebs/home/ubuntu /home/ubuntu
  
   sudo mkdir -p /mnt/ebs/var/lib/lxc
   sudo mkdir /var/lib/lxc
   sudo mount --bind /mnt/ebs/var/lib/lxc /var/lib/lxc

• Install Launchpad in an lxc container:

   cd
   sudo add-apt-repository ppa:yellow/ppa
   sudo add-apt-repository ppa:yellow/experimental
   sudo apt-get update
   sudo apt-get install lpsetup
   lp-setup lxc-install --testing --full-name="Benji York" --email="benji.york@canonical.com"

• Make a snapshot of the volume, make a new volume from the snapshot, make a new instance and attach the new volume to the instance.

   sudo apt-get install lxc
   sudo mkdir /mnt/ebs
   sudo mount /dev/xvdf /mnt/ebs
   sudo mkdir -p /mnt/ebs/home/ubuntu
   sudo mount --bind /mnt/ebs/home/ubuntu /home/ubuntu
   sudo mkdir -p /mnt/ebs/var/lib/lxc
   sudo mount --bind /mnt/ebs/var/lib/lxc /var/lib/lxc