Diff for "Running/LXC"

Not logged in - Log In / Register
Differences between revisions 67 and 69 (spanning 2 versions)
⇤ ← Revision 67 as of 2011-12-16 15:42:26 →
Size: 7526
Editor: gary
Comment:
← Revision 69 as of 2011-12-16 20:29:42 → ⇥
Size: 8623
Editor: gary
Comment:
Deletions are marked like this. Additions are marked like this.
Line 10: Line 10:

 1. Do not have dnsmasq installed (it is not by default). If you do, and you want to keep it, see the "Problems" section below for a workaround.
Line 25: Line 27:
 1. Create a container  1. Create a container. In the following commands, replace ''$username'' with your username on the host.
Line 27: Line 29:
sudo lxc-create -t ubuntu -n lucid-test-lp -f /etc/lxc/local.conf -- -r lucid -a i386 -b robertc sudo lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
Line 29: Line 31:
   If you want a proxy:
   {{{
sudo http_proxy=http://host:port/ lxc-create -t ubuntu -n lucid-test-lp -f /etc/lxc/local.conf -- -r lucid -a i386 -b robertc
    * gary_poster has used '''{{{-a i686}}}''' instead of '''{{{-a i386}}}''' on a 64 bit machine and it seems fine so far.

    * If you want a proxy:
 {{{
sudo http_proxy=http://host:port/ lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
Line 34: Line 39:
  And if you want to set a custom mirror, similar to http_proxy, but set MIRROR= instead.     * And if you want to set a custom mirror, similar to http_proxy, but set MIRROR= instead.

 1. ''(This is a workaround for [[Bug:902190|Bug 902190]]: see the "Updating" section.)'' At the bottom of '''{{{/var/lib/lxc/}}}''{{{$containername}}}''{{{/config}}}''' (e.g. '''{{{/var/lib/lxc/lpdev/config}}}''') add this snippet:
 {{{
# Mods to allow udev upgrade
lxc.cgroup.devices.allow = c 108:0 rwm
lxc.cgroup.devices.allow = b 7:0 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
}}}
Line 38: Line 51:
sudo lxc-start -n lucid-test-lp sudo lxc-start -n lpdev
Line 42: Line 55:
 1. You can log in as root (password "root"), or as your user, but without sudo powers (then use "su root" with password "root" to become root).  1. '''[Inside the container]''' Log in as root (password "root").
Line 44: Line 57:
 1. Grab the ip address (handed out via libvirt's dhcp server) - you may wish to ssh in rather than using the console (seems to have better termcap experience).     Alternatively, you can log in as your user, but without sudo powers (then use "su root" with password "root" to become root).

 1. '''[Inside the container]''' Grab the ip address (handed out via libvirt's dhcp server) - you may wish to ssh in rather than using the console (seems to have better termcap experience).
Line 48: Line 63:
    Alternatively, if you add '192.168.122.1' (libvirt's dnmasq default address) to your /etc/resolv.conf you can use:     Alternatively, if you add '192.168.122.1' (libvirt's dnsmasq default address) to your /etc/resolv.conf you can use:
Line 53: Line 68:
    [XXX Another alternative may be to use avahi. This should be tested and documented.  The advantage would be that you do not have to keep track of your vm's ipin resolv.conf as you change networks.]     [XXX Another alternative may be to use avahi. This should be tested and documented if desired.]
Line 55: Line 70:
 1. The new container won't have your proxy / mirror settings preserved. Customise it at this point before going further if you care about this.  1. '''[Inside the container]''' The new container won't have your proxy / mirror settings preserved. Customise it at this point before going further if you care about this.
Line 57: Line 72:
 1. Enable multiverse (rocketfuel-setup wants it, don't ask me why). See the "Updating" section below for more details on this.

 1. Install some additional packages we'll need to run rocketfuel-setup etc.
 {{{
apt-get install bzr
# if you have a localised (non-C) locale:
# not doing this will cause postgresql to fail to install, with -hilarious- results as database-developer-setup will think you have 8.2 installed.
# You can tell if you need this if the prior apt commands spewed locale warnings.
# Pick your specific language pack.
apt-get install language-pack-en
}}}

 1. Inside the container grant the user sudo rights:
 {{{
adduser $username sudo
}}}

 1. And add their user group:
 {{{
 addgroup --gid NNN $username
 }}}
 where NNN is as reported by
 {{{
 groups $username
 }}}
 
 1. To stop it now run 'poweroff -n' in the lxc instance and then 'sudo lxc-stop -n lucid-test-lp' in the host.
 
 1. Start it up again - headless now, we have the ip address from before.
 {{{
sudo lxc-start -n lucid-test-lp -d
}}}

 1. ssh <vm IP address> to connect to the VM. Your ssh key is already present because of the bind mount to your home dir.

 1. You can now follow the [[Getting|getting-started]] on LP instructions. Be warned that changes in ~ will affect you outside the container. You will want to run rocketfuel-setup with --no-workspace if your home already has a workarea. You may need to run utilities/launchpad-database-setup separately too.

 1. You probably want to follow [[Running/RemoteAccess]] has a discussion for how you can configure things so your non-container browser can access web pages from within the container.

 1. rabbitmq may fail to start up. If that happens it appears to be a [[http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-April/007024.html|mnesia glitch]] best sorted by zapping mnesia.
 {{{
sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
sudo service rabbit-mq start
}}}

= Updating =

Updating deb packages in an LXC has a few issues as of this writing.

 1. To get started, you'll need to add the apt sources. Here's an example /etc/apt/sources.list.		  1. '''[Inside the container]''' Enable multiverse (rocketfuel-setup wants it, don't ask me why). Here's an example /etc/apt/sources.list.
Line 118: Line 84:
 1. [[Bug:902190|Bug 902190]] describes the first problem when you will try to update. To work around this, at the bottom of "/var/lib/lxc/''server''/config" (e.g. /var/lib/lxc/lucid-test-lp/config) add this snippet:  1. '''[Inside the container]''' Install some additional packages we'll need to run rocketfuel-setup etc. Most people with an English locale will simply want to do this: '''{{{apt-get install bzr language-pack-en}}}'''

    If your locale is not English, or if you want more details, try/read this.
Line 120: Line 88:
# Mods to allow udev upgrade
lxc.cgroup.devices.allow = c 108:0 rwm
lxc.cgroup.devices.allow = b 7:0 rwm
lxc.cgroup.devices.allow = c 10:200 rwm		 apt-get install bzr
# if you have a localised (non-C) locale:
# not doing this will cause postgresql to fail to install, with -hilarious- results as database-developer-setup will think you have 8.2 installed.
# You can tell if you need this if the prior apt commands spewed locale warnings.
# Pick your specific language pack.
apt-get install language-pack-en
Line 126: Line 96:
 1. [[Bug:892892|Bug 892892]] describes the next problem you will encounter. Serge Hallyn reports that Stephane Graber has fixed this in Precise, and they will SRU it into Oneiric. [He suggests that for a quick fix, we can just turn off the devices cgroup - edit the container's config and comment out all 'lxc.cgroup.devices =' lines, but gary was unable to get this to work.] A quick workaround is to specify that you don't want to upgrade this package. I did it somehow with an aptitude gyration; documented approaches welcome.  1. '''[Inside the container]''' Grant the user sudo rights:
 {{{
adduser $username sudo
}}}

 1. '''[Inside the container]''' Add their user group:
 {{{
 addgroup --gid NNN $username
 }}}
 where NNN is as reported by
 {{{
 groups $username
 }}}
 
 1. To stop it now run 'poweroff' in the lxc container. If it works smoothly, you will eventually be dumped back out to your host system. If it looks like it is hanging, then use 'sudo lxc-stop -n lucid-test-lp' in the host.
 
 1. Start it up again - headless now, we have the ip address from before.
 {{{
sudo lxc-start -n lpdev -d
}}}

 1. ssh <vm IP address> to connect to the VM. Your ssh key is already present because of the bind mount to your home dir, though using ssh -A might give you a better ssh agent experience.

 1. You can now follow the [[Getting|getting-started]] on LP instructions. Be warned that changes in ~ will affect you outside the container. You will want to run rocketfuel-setup with --no-workspace if your home already has a workarea. You may need to run utilities/launchpad-database-setup separately too.

 1. You probably want to follow [[Running/RemoteAccess]] has a discussion for how you can configure things so your non-container browser can access web pages from within the container.


= Updating =

Updating deb packages in an LXC has a few issues as of this writing. One is already mentioned in the instructions above. This section records others, with workarounds.

 1. [[Bug:892892|Bug 892892]] describes a problem upgrading mountall. Serge Hallyn reports that Stephane Graber has fixed this in Precise, and they will SRU it into Oneiric. A quick workaround is to specify that you don't want to upgrade this package, such as with "=" in aptitude. [Serge suggests that for a quick fix, we can just turn off the devices cgroup - edit the container's config and comment out all 'lxc.cgroup.devices =' lines, but gary was unable to get this to work.]
Line 130: Line 132:
== rabbitmq does not start up ==

rabbitmq may fail to start up. If that happens it appears to be a [[http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2010-April/007024.html|mnesia glitch]] best sorted by zapping mnesia.
 {{{
sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
sudo service rabbit-mq start
}}}
Line 132: Line 142:
Gary encountered this problem when trying to contain a second lxc container. After trying more careful approaches, Serge Hallyn recommended that we just wipe out the lxc cache. "rm -rf /var/cache/lxc/*". This solved the problem. He went on to say that he wondered if we ought to automatically purge the cache every week or so, so the cache does seem potentially problematic. gary_poster encountered this problem when trying to contain a second lxc container. After trying more careful approaches, Serge Hallyn recommended that we just wipe out the lxc cache. "rm -rf /var/cache/lxc/*". This solved the problem. He went on to say that he wondered if we ought to automatically purge the cache every week or so, so the cache does seem potentially problematic.
Line 148: Line 158:
== lxc-start hangs ==

[[http://paste.ubuntu.com/772517/|The symptom looks like this]]. It hangs after that.

No fix or workaround identified yet, other than making a new lxc container.

To debug, try '''{{{lxc-start -n $containername -l debug -o outout}}}''' and look at outout.

This page explains how to set up and run Launchpad (for development) inside a LXC.

Why?

Launchpad development setup makes significant changes to your machine; its nice to be unaffected by those except when you are actually doing such development.

Also, launchpad has some limitations on concurrent testing per-machine and so forth - multiple container's can be used to work around this.

Make a LXC

  1. Do not have dnsmasq installed (it is not by default). If you do, and you want to keep it, see the "Problems" section below for a workaround.
  2. Install lxc 
    sudo apt-get install lxc libvirt-bin
  3. Create a config for your containers 
    sudo dd of=/etc/lxc/local.conf << EOF
lxc.network.type=veth
lxc.network.link=virbr0
lxc.network.flags=up
EOF

  4. Create a container. In the following commands, replace $username with your username on the host. 

    sudo lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username

    • gary_poster has used -a i686 instead of -a i386 on a 64 bit machine and it seems fine so far.

    • If you want a proxy: 
    sudo http_proxy=http://host:port/ lxc-create -t ubuntu -n lpdev -f /etc/lxc/local.conf -- -r lucid -a i386 -b $username
    • And if you want to set a custom mirror, similar to http_proxy, but set MIRROR= instead.

  5. (This is a workaround for Bug 902190: see the "Updating" section.) At the bottom of /var/lib/lxc/$containername/config (e.g. /var/lib/lxc/lpdev/config) add this snippet: 

    # Mods to allow udev upgrade
lxc.cgroup.devices.allow = c 108:0 rwm
lxc.cgroup.devices.allow = b 7:0 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
  6. Start the container 
    sudo lxc-start -n lpdev
    • Ignore the warning about openssh crashing - it restarts on a later event.

  7. [Inside the container] Log in as root (password "root").

    • Alternatively, you can log in as your user, but without sudo powers (then use "su root" with password "root" to become root).

  8. [Inside the container] Grab the ip address (handed out via libvirt's dhcp server) - you may wish to ssh in rather than using the console (seems to have better termcap experience). 

    ip addr show dev eth0 | grep 'inet'
    • Alternatively, if you add '192.168.122.1' (libvirt's dnsmasq default address) to your /etc/resolv.conf you can use: 
    ssh <container-name>.
    • [XXX Another alternative may be to use avahi. This should be tested and documented if desired.]

  9. [Inside the container] The new container won't have your proxy / mirror settings preserved. Customise it at this point before going further if you care about this.

  10. [Inside the container] Enable multiverse (rocketfuel-setup wants it, don't ask me why). Here's an example /etc/apt/sources.list. 

    deb http://archive.ubuntu.com/ubuntu lucid main universe
deb-src http://archive.ubuntu.com/ubuntu lucid main universe
deb http://archive.ubuntu.com/ubuntu lucid-updates main universe
deb-src http://archive.ubuntu.com/ubuntu lucid-updates main universe
deb http://archive.ubuntu.com/ubuntu lucid main multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid main multiverse
deb http://archive.ubuntu.com/ubuntu lucid-updates main multiverse
deb-src http://archive.ubuntu.com/ubuntu lucid-updates main multiverse

  11. [Inside the container] Install some additional packages we'll need to run rocketfuel-setup etc. Most people with an English locale will simply want to do this: apt-get install bzr language-pack-en

    • If your locale is not English, or if you want more details, try/read this. 
    apt-get install bzr
# if you have a localised (non-C) locale:
# not doing this will cause postgresql to fail to install, with -hilarious- results as database-developer-setup will think you have 8.2 installed.
# You can tell if you need this if the prior apt commands spewed locale warnings.
# Pick your specific language pack.
apt-get install language-pack-en

  12. [Inside the container] Grant the user sudo rights: 

    adduser $username sudo

  13. [Inside the container] Add their user group: 

     addgroup --gid NNN $username
    where NNN is as reported by 
     groups $username
  14. To stop it now run 'poweroff' in the lxc container. If it works smoothly, you will eventually be dumped back out to your host system. If it looks like it is hanging, then use 'sudo lxc-stop -n lucid-test-lp' in the host.
  15. Start it up again - headless now, we have the ip address from before. 
    sudo lxc-start -n lpdev -d

  16. ssh <vm IP address> to connect to the VM. Your ssh key is already present because of the bind mount to your home dir, though using ssh -A might give you a better ssh agent experience.

  17. You can now follow the getting-started on LP instructions. Be warned that changes in ~ will affect you outside the container. You will want to run rocketfuel-setup with --no-workspace if your home already has a workarea. You may need to run utilities/launchpad-database-setup separately too.

  18. You probably want to follow Running/RemoteAccess has a discussion for how you can configure things so your non-container browser can access web pages from within the container.

Updating

Updating deb packages in an LXC has a few issues as of this writing. One is already mentioned in the instructions above. This section records others, with workarounds.

  1. Bug 892892 describes a problem upgrading mountall. Serge Hallyn reports that Stephane Graber has fixed this in Precise, and they will SRU it into Oneiric. A quick workaround is to specify that you don't want to upgrade this package, such as with "=" in aptitude. [Serge suggests that for a quick fix, we can just turn off the devices cgroup - edit the container's config and comment out all 'lxc.cgroup.devices =' lines, but gary was unable to get this to work.]

Problems

rabbitmq does not start up

rabbitmq may fail to start up. If that happens it appears to be a mnesia glitch best sorted by zapping mnesia. 

  • sudo rm -rf /var/lib/rabbitmq/mnesia/rabbit/*
sudo service rabbit-mq start

lxc-create fails with errors that it is unable to access archive.ubuntu.com

gary_poster encountered this problem when trying to contain a second lxc container. After trying more careful approaches, Serge Hallyn recommended that we just wipe out the lxc cache. "rm -rf /var/cache/lxc/*". This solved the problem. He went on to say that he wondered if we ought to automatically purge the cache every week or so, so the cache does seem potentially problematic.

database-developer-setup fails, and thinks you are on Postgres 8.2

As noted above, if you have a localised (non-C) locale, you ned to install your specific language pack. For instance, if your computer has a localised English locale, use this: 

apt-get install language-pack-en

lxc-start fails, complaining that there is "No such device" of "virbr0"

Short answer: do you have dnsmasq installed? If so, uninstall it, or turn it off (/etc/init.d/dnsmasq stop). Then use sudo virsh net-start default and then retry. (Source)

Slightly more details: Normally, if you look at the "ifconfig" output, you will see a virbr0 interface. If it is not there, you'll have problems. In that case, you'll probably also see that the virtual network is inactive (see the output of virsh net-list --all). Apparently dnsmasq fights all this (and wins).

lxc-start hangs

The symptom looks like this. It hangs after that.

No fix or workaround identified yet, other than making a new lxc container.

To debug, try lxc-start -n $containername -l debug -o outout and look at outout.

Other problems

If other lxc users don't have an idea (known lxc users as of this writing include lifeless, wgrant, frankban and gary_poster) try asking hallyn or Spamaps on #ubuntu-server on freenode.

References

Alternatively

You can also run in a chroot environment or a VM.