Diff for "Running/LXC"

Not logged in - Log In / Register
Differences between revisions 4 and 87 (spanning 83 versions)
⇤ ← Revision 4 as of 2010-07-01 05:39:49 →
Size: 1498
Editor: lifeless
Comment: bzr whoami is needed.
← Revision 87 as of 2012-01-31 18:48:50 → ⇥
Size: 9395
Editor: bac
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
This page explains how to set up and run Launchpad (for development) inside a VM. This page explains how to set up and run Launchpad (for development) inside a LXC.
Line 5: Line 5:
Launchpad development setup makes numerous changes to your machine; its nice to be unaffected by those except when you are actually doing such development. Launchpad development setup makes significant changes to your machine; its nice to be unaffected by those except when you are actually doing such development.
Line 7: Line 7:
Also, launchpad has limitations on concurrent testing per-machine and so forth - multiple VM's can be used to work around this. Also, launchpad has some limitations on concurrent testing per-machine and so forth - multiple container's can be used to work around this.
Line 9: Line 9:
= Make a LXC =
Line 10: Line 11:
= Make a VM image =  1. Do not have dnsmasq installed (it is not by default). (dnsmasq-base is o.k. as it will be installed by the next step anyway.) If you do, and you want to keep it, see the "Problems" section below for some workarounds.
Line 12: Line 13:
 1. Install KVM  1. Install lxc
 {{{
sudo apt-get install lxc libvirt-bin
}}}

 1. Create a config for your containers
 {{{
sudo dd of=/etc/lxc/local.conf << EOF
lxc.network.type=veth
lxc.network.link=virbr0
lxc.network.flags=up
EOF
}}}

 1. Create a container. In the following commands, replace ''$username'' with your username on the host.
 {{{
sudo lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
}}}

    * Some people have used '''{{{-a i686}}}''' instead of '''{{{-a i386}}}''' on a 64 bit machine and it seems fine so far.

    * If you want a proxy:
 {{{
sudo http_proxy=http://host:port/ lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
}}}
 
    * And if you want to set a custom mirror, similar to http_proxy, but set MIRROR= instead.

 1. Start the container
 {{{
sudo lxc-start -n lpdev
}}}
    Ignore the warning about openssh crashing - it restarts on a later event.
    
 1. '''[Inside the container]''' Log in as root (password "root").

    Alternatively, you can log in as your user, but without sudo powers (then use "su root" with password "root" to become root).

 1. '''[Inside the container]''' Grab the ip address (handed out via libvirt's dhcp server) - you may wish to ssh in rather than using the console (seems to have better termcap experience).
 {{{
ip addr show dev eth0 | grep 'inet'
}}}
    Alternatively, if you add '192.168.122.1' (libvirt's dnsmasq default address) as the first nameserver in your /etc/resolv.conf you can use:
 {{{
ssh <container-name> (e.g. 'ssh lpdev')
}}}

    [XXX Another alternative may be to use avahi. This should be tested and documented if desired. {{{sudo apt-get install avahi-daemon}}} is the start...]

 1. '''[Inside the container]''' The new container won't have your proxy / mirror settings preserved. Customise it at this point before going further if you care about this.

 1. '''[Inside the container]''' Enable multiverse (rocketfuel-setup wants it, but no one has said why lately) and -updates. Here's an example minimal /etc/apt/sources.list.
 {{{
deb http://archive.ubuntu.com/ubuntu lucid main universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid-updates main universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid-security main universe multiverse
}}}
 If you are going to develop in this LXC you might very well want sources available. If so, add these too.
 {{{
deb-src http://archive.ubuntu.com/ubuntu lucid main universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-updates main universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-security main universe multiverse
}}}

 1. '''[Inside the container]''' Install some additional packages we'll need to run rocketfuel-setup etc. Most people with an English locale will simply want to do this: '''{{{apt-get install bzr language-pack-en}}}'''

    If your locale is not English, or if you want more details, try/read this.
 {{{
apt-get install bzr
# if you have a localised (non-C) locale:
# not doing this will cause postgresql to fail to install, with -hilarious- results as database-developer-setup will think you have 8.2 installed.
# You can tell if you need this if the prior apt commands spewed locale warnings.
# Pick your specific language pack.
apt-get install language-pack-en
}}}

 1. '''[Inside the container]''' Grant the user sudo rights:
 {{{
adduser $username sudo
}}}

 1. '''[Inside the container]''' Add their user group:
 {{{
 addgroup --gid NNN $username
 }}}
 where NNN is as reported by
 {{{
 groups $username
 }}}
 
 1. To stop it now run 'poweroff' in the lxc container. If it works smoothly, you will eventually be dumped back out to your host system. If it looks like it is hanging, then use "{{{sudo lxc-stop -n lpdev}}}" in the host.
 
 1. Start it up again, headless this time (-d). The previous IP address will be used.
 {{{
sudo lxc-start -n lpdev -d
}}}

 1. ssh <vm IP address> to connect to the VM. Your ssh key is already present because of the bind mount to your home dir, though using ssh -A might give you a better ssh agent experience.

 1. You can now follow the [[Getting|getting-started]] on LP instructions. Be warned that changes in ~ will affect you outside the container. You will want to run rocketfuel-setup with --no-workspace if your home already has a workarea. You may need to run utilities/launchpad-database-setup separately too.

 1. You probably want to follow [[Running/RemoteAccess]] has a discussion for how you can configure things so your non-container browser can access web pages from within the container.

= Problems =

== rabbitmq does not start up ==

rabbitmq may fail to start up. If that happens it appears to be a [[http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-April/007024.html|mnesia glitch]] best sorted by zapping mnesia.
 {{{
sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
sudo service rabbit-mq start
}}}

== updating doesn't work ==

Updating deb packages with apt or aptitude in an LXC has a few issues as of this writing. One is already mentioned in the instructions above. This section records others, with workarounds.

[[Bug:892892|Bug 892892]] describes a problem upgrading mountall. Serge Hallyn reports that Stephane Graber has fixed this in Precise, and they will SRU it into Oneiric. A quick workaround is to specify that you don't want to upgrade this package, such as with "=" in aptitude. [Serge suggests that for a quick fix, we can just turn off the devices cgroup - edit the container's config (e.g. `/var/lib/lxc/lpdev/config`) and comment out all `lxc.cgroup.devices =` lines, but gary was unable to get this to work.]

== lxc-create fails with errors that it is unable to access archive.ubuntu.com ==

See [[Bug:906500| bug 906500]]. gary_poster encountered this problem when trying to create a second lxc container. After trying more careful approaches to a solution, Serge Hallyn recommended that we just wipe out the lxc cache. "rm -rf /var/cache/lxc/*". This made the problem go away.

== database-developer-setup fails, and thinks you are on Postgres 8.2 ==

As noted above, if you have a localised (non-C) locale, you need to install your specific language pack. For instance, if your computer has a localised English locale, use this:
Line 15: Line 141:
% sudo apt-get install virt-manager apt-get install language-pack-en
Line 18: Line 144:
 1. Download the Lucid server ISO == lxc-start fails, complaining that there is "No such device" of "virbr0" ==
Line 20: Line 146:
 1. Run virt-manager. Do you have dnsmasq installed? If so, uninstall it, or do one of these two workarounds.
Line 22: Line 148:
 1. Double click on localhost(QEMU)  1. Perhaps [[http://wiki.libvirt.org/page/Libvirtd_and_dnsmasq|this workaround]] will do the trick for you. If you have success, please record it here.
Line 24: Line 150:
 1. click on the New virtual machine icon  1. Turn it off when you need lxc ('''{{{/etc/init.d/dnsmasq stop}}}'''). Then use '''{{{sudo virsh net-start default}}}''' and then retry. [[http://mytipsandtricson.blogspot.com/2010/12/kvm-failed-to-start-network-default-in.html|(Source)]]
Line 26: Line 152:
 1. follow your nose here, using the ISO as the install media, and allocating no less than 2G of disk and 1G of memory. I suggest 4G if you can spare it. Slightly more details: Normally, if you look at the "ifconfig" output in the host, you will see a virbr0 interface. If it is not there, you'll have problems. In that case, you'll probably also see that the virtual network is inactive (see the output of '''{{{virsh net-list --all}}}''').
Line 28: Line 154:
 1. After its installed, connect to the image and install {{{acpid}}} and {{{openssh-server}}} == lxc-start hangs ==
Line 30: Line 156:
 1. Use ssh-copy-id to copy your public key into the VM. [[http://paste.ubuntu.com/772517/|The symptom looks like this]]. It hangs after that.
Line 32: Line 158:
 1. ssh -A <vm IP address> to connect to the VM. No fix or workaround identified yet, other than making a new lxc container.
Line 34: Line 160:
 1. {{{bzr whoami "Your Name <your.email@example.com>"}}} to set your bzr identity in the VM. To debug, try '''{{{lxc-start -n $containername -l debug -o outout}}}''' and look at outout.
Line 36: Line 162:
 1. You can now follow the [[Getting|getting-started]] on LP instructions. == DNS fails inside the container ==

After restarting in daemon mode and logging in as a regular user, DNS was not working.
Ensure there is a nameserver in the container's /etc/resolv.conf, which is created at startup by resolverconf. Stopping and starting the container solved the problem.

== Random flakiness ==

Using lxc via juju I ran into all sorts of problems with DNS, version mismatches, etc. Since it was via juju I wasn't able to muck around with /etc/resolv.conf (the damage was done before I got the chance to ssh to the guest.) I found {{{sudo rm -rf /var/cache/lxc}}} solved the problem. It is rather brutal but worked. Of course the next run took a long time as all of that previously cached stuff had to be refetched.

== Other problems ==

If other lxc users don't have an idea (known lxc users as of this writing include lifeless, wgrant, frankban and gary_poster) try asking hallyn or Spamaps on #ubuntu-server on freenode.
Line 40: Line 177:
See also this email thread about [[https://lists.launchpad.net/launchpad-dev/msg03456.html|running Launchpad in a virtual machine]], and this [[https://lists.launchpad.net/launchpad-dev/msg03454.html|discussion of the differences]] between running in a [[Running/Schroot|chroot]] environment and running a VM.
= Alternatively =

You can also run in a [[Running/Schroot|chroot]] environment or a [[Running/VirtualMachine|VM]].

This page explains how to set up and run Launchpad (for development) inside a LXC.

Why?

Launchpad development setup makes significant changes to your machine; its nice to be unaffected by those except when you are actually doing such development.

Also, launchpad has some limitations on concurrent testing per-machine and so forth - multiple container's can be used to work around this.

Make a LXC

  1. Do not have dnsmasq installed (it is not by default). (dnsmasq-base is o.k. as it will be installed by the next step anyway.) If you do, and you want to keep it, see the "Problems" section below for some workarounds.
  2. Install lxc 
    sudo apt-get install lxc libvirt-bin
  3. Create a config for your containers 
    sudo dd of=/etc/lxc/local.conf << EOF
lxc.network.type=veth
lxc.network.link=virbr0
lxc.network.flags=up
EOF

  4. Create a container. In the following commands, replace $username with your username on the host. 

    sudo lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username

    • Some people have used -a i686 instead of -a i386 on a 64 bit machine and it seems fine so far.

    • If you want a proxy: 
    sudo http_proxy=http://host:port/ lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
    • And if you want to set a custom mirror, similar to http_proxy, but set MIRROR= instead.
  5. Start the container 
    sudo lxc-start -n lpdev
    • Ignore the warning about openssh crashing - it restarts on a later event.

  6. [Inside the container] Log in as root (password "root").

    • Alternatively, you can log in as your user, but without sudo powers (then use "su root" with password "root" to become root).

  7. [Inside the container] Grab the ip address (handed out via libvirt's dhcp server) - you may wish to ssh in rather than using the console (seems to have better termcap experience). 

    ip addr show dev eth0 | grep 'inet'
    • Alternatively, if you add '192.168.122.1' (libvirt's dnsmasq default address) as the first nameserver in your /etc/resolv.conf you can use: 
    ssh <container-name> (e.g. 'ssh lpdev')

    • [XXX Another alternative may be to use avahi. This should be tested and documented if desired. sudo apt-get install avahi-daemon is the start...]

  8. [Inside the container] The new container won't have your proxy / mirror settings preserved. Customise it at this point before going further if you care about this.

  9. [Inside the container] Enable multiverse (rocketfuel-setup wants it, but no one has said why lately) and -updates. Here's an example minimal /etc/apt/sources.list. 

    deb http://archive.ubuntu.com/ubuntu lucid main universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid-updates main universe multiverse
deb http://archive.ubuntu.com/ubuntu lucid-security main universe multiverse
    If you are going to develop in this LXC you might very well want sources available. If so, add these too. 
    deb-src http://archive.ubuntu.com/ubuntu lucid main universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-updates main universe multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-security main universe multiverse

  10. [Inside the container] Install some additional packages we'll need to run rocketfuel-setup etc. Most people with an English locale will simply want to do this: apt-get install bzr language-pack-en

    • If your locale is not English, or if you want more details, try/read this. 
    apt-get install bzr
# if you have a localised (non-C) locale:
# not doing this will cause postgresql to fail to install, with -hilarious- results as database-developer-setup will think you have 8.2 installed.
# You can tell if you need this if the prior apt commands spewed locale warnings.
# Pick your specific language pack.
apt-get install language-pack-en

  11. [Inside the container] Grant the user sudo rights: 

    adduser $username sudo

  12. [Inside the container] Add their user group: 

     addgroup --gid NNN $username
    where NNN is as reported by 
     groups $username

  13. To stop it now run 'poweroff' in the lxc container. If it works smoothly, you will eventually be dumped back out to your host system. If it looks like it is hanging, then use "sudo lxc-stop -n lpdev" in the host.

  14. Start it up again, headless this time (-d). The previous IP address will be used. 
    sudo lxc-start -n lpdev -d

  15. ssh <vm IP address> to connect to the VM. Your ssh key is already present because of the bind mount to your home dir, though using ssh -A might give you a better ssh agent experience.

  16. You can now follow the getting-started on LP instructions. Be warned that changes in ~ will affect you outside the container. You will want to run rocketfuel-setup with --no-workspace if your home already has a workarea. You may need to run utilities/launchpad-database-setup separately too.

  17. You probably want to follow Running/RemoteAccess has a discussion for how you can configure things so your non-container browser can access web pages from within the container.

Problems

rabbitmq does not start up

rabbitmq may fail to start up. If that happens it appears to be a mnesia glitch best sorted by zapping mnesia. 

  • sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
sudo service rabbit-mq start

updating doesn't work

Updating deb packages with apt or aptitude in an LXC has a few issues as of this writing. One is already mentioned in the instructions above. This section records others, with workarounds.

Bug 892892 describes a problem upgrading mountall. Serge Hallyn reports that Stephane Graber has fixed this in Precise, and they will SRU it into Oneiric. A quick workaround is to specify that you don't want to upgrade this package, such as with "=" in aptitude. [Serge suggests that for a quick fix, we can just turn off the devices cgroup - edit the container's config (e.g. /var/lib/lxc/lpdev/config) and comment out all lxc.cgroup.devices = lines, but gary was unable to get this to work.]

lxc-create fails with errors that it is unable to access archive.ubuntu.com

See bug 906500. gary_poster encountered this problem when trying to create a second lxc container. After trying more careful approaches to a solution, Serge Hallyn recommended that we just wipe out the lxc cache. "rm -rf /var/cache/lxc/*". This made the problem go away.

database-developer-setup fails, and thinks you are on Postgres 8.2

As noted above, if you have a localised (non-C) locale, you need to install your specific language pack. For instance, if your computer has a localised English locale, use this: 

apt-get install language-pack-en

lxc-start fails, complaining that there is "No such device" of "virbr0"

Do you have dnsmasq installed? If so, uninstall it, or do one of these two workarounds.

  1. Perhaps this workaround will do the trick for you. If you have success, please record it here.

  2. Turn it off when you need lxc (/etc/init.d/dnsmasq stop). Then use sudo virsh net-start default and then retry. (Source)

Slightly more details: Normally, if you look at the "ifconfig" output in the host, you will see a virbr0 interface. If it is not there, you'll have problems. In that case, you'll probably also see that the virtual network is inactive (see the output of virsh net-list --all).

lxc-start hangs

The symptom looks like this. It hangs after that.

No fix or workaround identified yet, other than making a new lxc container.

To debug, try lxc-start -n $containername -l debug -o outout and look at outout.

DNS fails inside the container

After restarting in daemon mode and logging in as a regular user, DNS was not working. Ensure there is a nameserver in the container's /etc/resolv.conf, which is created at startup by resolverconf. Stopping and starting the container solved the problem.

Random flakiness

Using lxc via juju I ran into all sorts of problems with DNS, version mismatches, etc. Since it was via juju I wasn't able to muck around with /etc/resolv.conf (the damage was done before I got the chance to ssh to the guest.) I found sudo rm -rf /var/cache/lxc solved the problem. It is rather brutal but worked. Of course the next run took a long time as all of that previously cached stuff had to be refetched.

Other problems

If other lxc users don't have an idea (known lxc users as of this writing include lifeless, wgrant, frankban and gary_poster) try asking hallyn or Spamaps on #ubuntu-server on freenode.

References

Alternatively

You can also run in a chroot environment or a VM.