Diff for "Running/LXC"

Not logged in - Log In / Register

Differences between revisions 84 and 112 (spanning 28 versions)
Revision 84 as of 2012-01-10 16:42:21
Size: 8762
Editor: bac
Comment:
Revision 112 as of 2017-09-22 08:30:24
Size: 4165
Editor: cjwatson
Comment: production is on 16.04 now
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
This page explains how to set up and run Launchpad (for development) inside a LXC. This page explains how to set up and run Launchpad (for development) inside an LXC container. LXC is the recommended environment for doing Launchpad development in; the core Launchpad team and CI infrastructure all use LXC extensively. If your setup permits it, you should use [[Running/LXD]] instead, which is the recommended setup and what most developers currently use.
Line 5: Line 5:
Launchpad development setup makes significant changes to your machine; its nice to be unaffected by those except when you are actually doing such development. Launchpad development setup makes significant changes to your machine; it's nice to be unaffected by those when you're not doing such development. Also, multiple containers can be used to work around Launchpad's limitations regarding concurrent test runs on a single machine.
Line 7: Line 7:
Also, launchpad has some limitations on concurrent testing per-machine and so forth - multiple container's can be used to work around this. These instructions should work Ubuntu 14.04 LTS or later, and with some adaptation on any Linux distribution that uses LXC 1.0.0 or later. Older versions of LXC are significantly less reliable and polished, so we recommend upgrading to 14.04 LTS and removing `/var/cache/lxc` to ensure a clean, working start.
Line 9: Line 9:
= Make a LXC = = Create an LXC container =
Line 11: Line 11:
 1. Do not have dnsmasq installed (it is not by default). (dnsmasq-base is o.k. as it will be installed by the next step anyway.) If you do, and you want to keep it, see the "Problems" section below for some workarounds.

 1. Install lxc
 1. Install LXC's userspace tools.
Line 15: Line 13:
sudo apt-get install lxc libvirt-bin sudo apt-get install lxc
Line 18: Line 16:
 1. Create a config for your containers  1. Create a container. You can use an HTTP proxy or alternate Ubuntu mirror by specifying an http_proxy or MIRROR environment variable after `sudo`. (amd64 works fine too, although it will use more RAM.)
Line 20: Line 18:
sudo dd of=/etc/lxc/local.conf << EOF
lxc.network.type=veth
lxc.network.link=virbr0
lxc.network.flags=up
EOF
sudo lxc-create -t ubuntu -n lpdev -- -r xenial -a i386 -b $USER
Line 27: Line 21:
 1. Create a container. In the following commands, replace ''$username'' with your username on the host.
 {{{
sudo lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
}}}

    * Some people have used '''{{{-a i686}}}''' instead of '''{{{-a i386}}}''' on a 64 bit machine and it seems fine so far.

    * If you want a proxy:
 {{{
sudo http_proxy=http://host:port/ lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
}}}
 
    * And if you want to set a custom mirror, similar to http_proxy, but set MIRROR= instead.

 1. Start the container
 {{{
sudo lxc-start -n lpdev
}}}
    Ignore the warning about openssh crashing - it restarts on a later event.
    
 1. '''[Inside the container]''' Log in as root (password "root").

    Alternatively, you can log in as your user, but without sudo powers (then use "su root" with password "root" to become root).

 1. '''[Inside the container]''' Grab the ip address (handed out via libvirt's dhcp server) - you may wish to ssh in rather than using the console (seems to have better termcap experience).
 {{{
ip addr show dev eth0 | grep 'inet'
}}}
    Alternatively, if you add '192.168.122.1' (libvirt's dnsmasq default address) as the first nameserver in your /etc/resolv.conf you can use:
 {{{
ssh <container-name> (e.g. 'ssh lpdev')
}}}

    [XXX Another alternative may be to use avahi. This should be tested and documented if desired. {{{sudo apt-get install avahi-daemon}}} is the start...]

 1. '''[Inside the container]''' The new container won't have your proxy / mirror settings preserved. Customise it at this point before going further if you care about this.

 1. '''[Inside the container]''' Enable multiverse (rocketfuel-setup wants it, but no one has said why lately) and -updates. Here's an example minimal /etc/apt/sources.list.
 {{{
deb http://archive.ubuntu.com/ubuntu lucid main universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid-updates main universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid-security main universe multiverse
}}}
 If you are going to develop in this LXC you might very well want sources available. If so, add these too.
 {{{
deb-src http://archive.ubuntu.com/ubuntu lucid main universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-updates main universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-security main universe multiverse
}}}

 1. '''[Inside the container]''' Install some additional packages we'll need to run rocketfuel-setup etc. Most people with an English locale will simply want to do this: '''{{{apt-get install bzr language-pack-en}}}'''

    If your locale is not English, or if you want more details, try/read this.
 {{{
apt-get install bzr
# if you have a localised (non-C) locale:
# not doing this will cause postgresql to fail to install, with -hilarious- results as database-developer-setup will think you have 8.2 installed.
# You can tell if you need this if the prior apt commands spewed locale warnings.
# Pick your specific language pack.
apt-get install language-pack-en
}}}

 1. '''[Inside the container]''' Grant the user sudo rights:
 {{{
adduser $username sudo
}}}

 1. '''[Inside the container]''' Add their user group:
 {{{
 addgroup --gid NNN $username
 }}}
 where NNN is as reported by
 {{{
 groups $username
 }}}
 
 1. To stop it now run 'poweroff' in the lxc container. If it works smoothly, you will eventually be dumped back out to your host system. If it looks like it is hanging, then use "{{{sudo lxc-stop -n lpdev}}}" in the host.
 
 1. Start it up again, headless this time (-d). The previous IP address will be used.
 1. Start the container in the background.
Line 110: Line 26:
 1. ssh <vm IP address> to connect to the VM. Your ssh key is already present because of the bind mount to your home dir, though using ssh -A might give you a better ssh agent experience.  1. Find its IP address.
 {{{
sudo lxc-ls --fancy
}}}
    
 1. `ssh -A IP_ADDRESS_FROM_LXC_LS` to connect to the container. Your username and password will match your account on the host system. If your SSH key is in your local `authorized_keys` file you shouldn't be prompted for a password, as your home directory (including public and private keys) is bind mounted into the container. The `-A` permits you to access Launchpad code hosting from within the container without needing to reenter passphrases.
Line 112: Line 33:
 1. You can now follow the [[Getting|getting-started]] on LP instructions. Be warned that changes in ~ will affect you outside the container. You will want to run rocketfuel-setup with --no-workspace if your home already has a workarea. You may need to run utilities/launchpad-database-setup separately too.  1. '''[Inside the container]''' Install Bazaar so you can run `rocketfuel-setup`.
 {{{
 sudo apt-get install bzr
}}}
Line 114: Line 38:
 1. You probably want to follow [[Running/RemoteAccess]] has a discussion for how you can configure things so your non-container browser can access web pages from within the container.  1. '''[Inside the container]''' You can now follow the normal [[Running|LP installation instructions]]. Be warned that changes in your home directory will also be seen outside the container and vice versa. If your home directory already has a Launchpad work area set up you'll want to run `rocketfuel-setup --no-workspace` to avoid trying to recreate it, but all subsequent steps are still required.
Line 116: Line 40:
= Problems =  1. Follow [[Running/RemoteAccess]] to set up access from the host's applications to the container's Launchpad instance.
Line 118: Line 42:
== rabbitmq does not start up == == Assigning a static IP to the container ==
Line 120: Line 44:
rabbitmq may fail to start up. If that happens it appears to be a [[http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-April/007024.html|mnesia glitch]] best sorted by zapping mnesia.
 {{{
sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
sudo service rabbit-mq start
If you would like to assign a static IP to the container via dhcp:

  1. In `/etc/default/lxc-net` ensure the following lines are uncommented:
  {{{
LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf
LXC_DOMAIN="lxc"
Line 125: Line 51:
  2. Create `/etc/lxc/dnsmasq.conf` and add the following line (change IP to suit):
  {{{
dhcp-host=lpdev,10.0.3.210
}}}
  3. Stop '''all''' lxc containers (lxc-net may not correctly restart if any containers are running):
  {{{
sudo lxc-stop --name lpdev
}}}
  4. Run `sudo service lxc-net restart` and start the container.
Line 126: Line 61:
== updating doesn't work == The lpdev container should restart with the specified IP.
Line 128: Line 63:
Updating deb packages with apt or aptitude in an LXC has a few issues as of this writing. One is already mentioned in the instructions above. This section records others, with workarounds. Alternatively, a static IP can be configured directly in the container from `/var/lib/lxc/lpdev/rootfs/etc/network/interfaces`.
Line 130: Line 65:
[[Bug:892892|Bug 892892]] describes a problem upgrading mountall. Serge Hallyn reports that Stephane Graber has fixed this in Precise, and they will SRU it into Oneiric. A quick workaround is to specify that you don't want to upgrade this package, such as with "=" in aptitude. [Serge suggests that for a quick fix, we can just turn off the devices cgroup - edit the container's config and comment out all 'lxc.cgroup.devices =' lines, but gary was unable to get this to work.] = Troubleshooting =
Line 132: Line 67:
== lxc-create fails with errors that it is unable to access archive.ubuntu.com == <<Anchor(postgresql-locale-breakage)>>
== launchpad-database-setup fails ==
Line 134: Line 70:
See [[Bug:906500| bug 906500]]. gary_poster encountered this problem when trying to create a second lxc container. After trying more careful approaches to a solution, Serge Hallyn recommended that we just wipe out the lxc cache. "rm -rf /var/cache/lxc/*". This made the problem go away. PostgreSQL will fail to create a cluster during installation if your
locale is configured to something non-C but not supported by the
container, so you need to install the relevant language pack.
Line 136: Line 74:
== database-developer-setup fails, and thinks you are on Postgres 8.2 == Modern LXC container templates do this automatically, but you will know
you need to do it manually if bzr or apt commands have been spewing locale
warnings.
Line 138: Line 78:
As noted above, if you have a localised (non-C) locale, you need to install your specific language pack. For instance, if your computer has a localised English locale, use this: For instance, if your computer has a localised English locale, use this:
Line 144: Line 84:
== lxc-start fails, complaining that there is "No such device" of "virbr0" ==

Do you have dnsmasq installed? If so, uninstall it, or do one of these two workarounds.

 1. Perhaps [[http://wiki.libvirt.org/page/Libvirtd_and_dnsmasq|this workaround]] will do the trick for you. If you have success, please record it here.

 1. Turn it off when you need lxc ('''{{{/etc/init.d/dnsmasq stop}}}'''). Then use '''{{{sudo virsh net-start default}}}''' and then retry. [[http://mytipsandtricson.blogspot.com/2010/12/kvm-failed-to-start-network-default-in.html|(Source)]]

Slightly more details: Normally, if you look at the "ifconfig" output in the host, you will see a virbr0 interface. If it is not there, you'll have problems. In that case, you'll probably also see that the virtual network is inactive (see the output of '''{{{virsh net-list --all}}}''').

== lxc-start hangs ==

[[http://paste.ubuntu.com/772517/|The symptom looks like this]]. It hangs after that.

No fix or workaround identified yet, other than making a new lxc container.

To debug, try '''{{{lxc-start -n $containername -l debug -o outout}}}''' and look at outout.

== DNS fails inside the container ==

After restarting in daemon mode and logging in as a regular user, DNS was not working.
No current resolution.

== Other problems ==

If other lxc users don't have an idea (known lxc users as of this writing include lifeless, wgrant, frankban and gary_poster) try asking hallyn or Spamaps on #ubuntu-server on freenode.

= References =


= Alternatively =

You can also run in a [[Running/Schroot|chroot]] environment or a [[Running/VirtualMachine|VM]].
If you didn't install the language pack before running rocketfuel-setup,
you'll need to run `sudo pg_createcluster 9.3 main` afterwards to fix
the damage.

This page explains how to set up and run Launchpad (for development) inside an LXC container. LXC is the recommended environment for doing Launchpad development in; the core Launchpad team and CI infrastructure all use LXC extensively. If your setup permits it, you should use Running/LXD instead, which is the recommended setup and what most developers currently use.

Why?

Launchpad development setup makes significant changes to your machine; it's nice to be unaffected by those when you're not doing such development. Also, multiple containers can be used to work around Launchpad's limitations regarding concurrent test runs on a single machine.

These instructions should work Ubuntu 14.04 LTS or later, and with some adaptation on any Linux distribution that uses LXC 1.0.0 or later. Older versions of LXC are significantly less reliable and polished, so we recommend upgrading to 14.04 LTS and removing /var/cache/lxc to ensure a clean, working start.

Create an LXC container

  1. Install LXC's userspace tools.
    sudo apt-get install lxc
  2. Create a container. You can use an HTTP proxy or alternate Ubuntu mirror by specifying an http_proxy or MIRROR environment variable after sudo. (amd64 works fine too, although it will use more RAM.)

    sudo lxc-create -t ubuntu -n lpdev -- -r xenial -a i386 -b $USER
  3. Start the container in the background.
    sudo lxc-start -n lpdev -d
  4. Find its IP address.
    sudo lxc-ls --fancy
  5. ssh -A IP_ADDRESS_FROM_LXC_LS to connect to the container. Your username and password will match your account on the host system. If your SSH key is in your local authorized_keys file you shouldn't be prompted for a password, as your home directory (including public and private keys) is bind mounted into the container. The -A permits you to access Launchpad code hosting from within the container without needing to reenter passphrases.

  6. [Inside the container] Install Bazaar so you can run rocketfuel-setup.

     sudo apt-get install bzr
  7. [Inside the container] You can now follow the normal LP installation instructions. Be warned that changes in your home directory will also be seen outside the container and vice versa. If your home directory already has a Launchpad work area set up you'll want to run rocketfuel-setup --no-workspace to avoid trying to recreate it, but all subsequent steps are still required.

  8. Follow Running/RemoteAccess to set up access from the host's applications to the container's Launchpad instance.

Assigning a static IP to the container

If you would like to assign a static IP to the container via dhcp:

  1. In /etc/default/lxc-net ensure the following lines are uncommented:

    LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf
    LXC_DOMAIN="lxc"
  2. Create /etc/lxc/dnsmasq.conf and add the following line (change IP to suit):

    dhcp-host=lpdev,10.0.3.210
  3. Stop all lxc containers (lxc-net may not correctly restart if any containers are running):

    sudo lxc-stop --name lpdev
  4. Run sudo service lxc-net restart and start the container.

The lpdev container should restart with the specified IP.

Alternatively, a static IP can be configured directly in the container from /var/lib/lxc/lpdev/rootfs/etc/network/interfaces.

Troubleshooting

launchpad-database-setup fails

PostgreSQL will fail to create a cluster during installation if your locale is configured to something non-C but not supported by the container, so you need to install the relevant language pack.

Modern LXC container templates do this automatically, but you will know you need to do it manually if bzr or apt commands have been spewing locale warnings.

For instance, if your computer has a localised English locale, use this:

apt-get install language-pack-en

If you didn't install the language pack before running rocketfuel-setup, you'll need to run sudo pg_createcluster 9.3 main afterwards to fix the damage.