5595
Comment: Add NTP server configuration
|
5811
get build slave installation instructions a bit closer to production
|
Deletions are marked like this. | Additions are marked like this. |
Line 1: | Line 1: |
You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side {{{launchpad-buildd}}} package on your system. |
|
Line 3: | Line 5: |
* Alter the following sections of the configuration (`configs/development/launchpad-lazr.conf`) as shown: | * Run {{{utilities/start-dev-soyuz.sh}}} to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory. * Once you've set up your test database, run {{{utilities/soyuz-sampledata-setup.py -e you@example.com}}} (where ''you@example.com'' should be an email address you own and have a GPG key for). This prepares more suitable sample data in the {{{launchpad_dev}}} database, including recent Ubuntu series. If you get a "duplicate key" error, {{{make schema}}} and run again. * `make run` (or if you also want to use codehosting, `make run_codehosting`—some services may fail to start up because you already started them, but it shouldn't be a problem). * Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password ''test''. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member {{{ubuntu-team}}} (conferring upload rights to the primary archive). == Extra PPA dependencies == The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want: * Log in as `admin@canonical.com:test` (I suggest using a different browser so you don't break up your ongoing session). * Open https://launchpad.dev/~ppa-user/+archive/test-ppa/+admin * Edit external dependencies. They normally look like: {{{ deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse }}} = Setup a build slave = == Installation == * Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set `lxc.aa_profile = unconfined` in `/var/lib/lxc/container-name/config` which is required to disable App``Armor support. If you are running Launchpad in a container, you will more than likely want your VM's network bridged on `lxcbr0`. In your slave vm/lxc: * `sudo apt-add-repository ppa:launchpad/buildd-staging` * `sudo apt-get update` * `sudo apt-get install launchpad-buildd bzr-builder quilt binfmt-support qemu-user-static` Alternatively, launchpad-buildd can be built from `lp:launchpad-buildd` with `dpkg-buildpackage -b`. * Edit `/etc/launchpad-buildd/default` and make sure ntphost points to an existing NTP server. You can check the [[http://www.pool.ntp.org/|NTP server pool]] to find one near you. To run the build slave by default, you should make sure that other hosts on the Internet cannot send requests to it! Then: * `echo RUN_NETWORK_REQUESTS_AS_ROOT=yes > ./etc/default/launchpad-buildd` == Launchpad Configuration == From your host system: * Get an Ubuntu buildd chroot from Launchpad, using `manage-chroot` from [[https://code.launchpad.net/+branch/ubuntu-archive-tools|lp:ubuntu-archive-tools]]: * `manage-chroot -s precise -a i386 get` * `LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set` * Point `url` to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit) Bob, shortly thereafter should report a successful status of 'idle'. = Drive slave through rpc = With librarian running, fire up a python shell and: |
Line 6: | Line 62: |
[archivepublisher] root: /var/tmp/archive base_url: http://archive.launchpad.dev/ # <- this [builddmaster] root: /var/tmp/builddmaster/ uploader: scripts/process-upload.py -Mvv --context buildd # <- and this |
import xmlrpclib proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc') proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '') proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {}, {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'}) proxy.status() proxy.clean() # Clean up if it failed |
Line 15: | Line 71: |
* In /etc/hosts, point `archive.launchpad.dev` at 127.0.0.88 like the rest. * Add to `/etc/apache2/site-available/local-launchpad`: {{{ <VirtualHost 127.0.0.88:80> ServerName archive.launchpad.dev LogLevel debug DocumentRoot /var/tmp/archive <Directory /var/tmp/archive/> Order Deny,Allow Deny from all Allow from 127.0.0.0/255.0.0.0 Options Indexes </Directory> </VirtualHost> }}} * Grab and run [[attachment:make-ubuntu-sane.py|make-ubuntu-sane.py]], to set up some Ubuntu series that still exist, along with some state such as permissions. * Grab and run [[attachment:start-soyuz.sh|start-soyuz.sh]], which will fire up all Soyuz services. This will eat lots of RAM. = Configure an account and PPA = * `make run` * Create an account for yourself (`utilities/make-lp-user USERNAME`) * Log in with `USERNAME@example.com:test`. * Add your OpenPGP key * `gpg --keyserver keyserver.launchpad.dev --send-keys DEADBEEF` * Add it to your Launchpad account, as normal. * Check email in `root`'s mailbox, and use it to activate the key. * Sign the code of conduct. * Create a PPA. = Set up the PPA = * Log out. * Log in as `admin@example.com:test`. * Browse to the new PPA. * Click "Administer". * Ensure that the PPA is allowed to build on non-virtual builders. * Add an external dependency so you don't have to import the entire Ubuntu archive just for a PPA to work: {{{ deb http://archive.ubuntu.com/ubuntu %(series) main restricted universe multiverse }}} = Configure a buildd = * `cd lib/canonical/buildd` * `dpkg-buildpackage -b` * `sudo dpkg -i ../launchpad-buildd_50_all.deb` * Make it work. * Add `$SUDO cp /etc/resolv.conf /etc/hosts "$HOME/build-$BUILDID/chroot-autobuild/etc"` to the end of `/usr/share/launchpad-buildd/slavebin/mount-chroot`, so name resolution works. * Might need to add `--umask=022` to `twistd` args in `/etc/init.d/launchpad-buildd`, or `pkg-create-dbgsym` fails obscurely. * (>= Karmic) Might need to alter "`dpkg-source: extracting`" output grepping in sbuild to "`dpkg-source: info: extracting`" * Edit `/etc/launchpad-buildd/default` and make sure `ntphost` points to an existing NTP server. You can check the [[http://www.pool.ntp.org/|NTP server pool]] to find one near you. * Get an Ubuntu buildd chroot. * Look for the latest build of your favourite architecture and series at https://launchpad.net/ubuntu/+builds. * Find the chroot SHA1 in the third line of the log. * Search for it at http://launchpadlibrarian.net/search?digest=SOMESHA1 * Grab one of the listed URLs. * http://launchpadlibrarian.net/34617356/chroot-ubuntu-lucid-i386.tar.bz2 is the current lucid i386 chroot at the time of writing. * `scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 add -f chroot-ubuntu-lucid-i386.tar.bz2` * Mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit) Please note a "builder" and a "chroot" are not the same thing, nor are they "associated" in any way. The logic is as follows: * PPAs require builders (a machine where to build packages) * The builder may be virtualized (for instance, launchpad.net uses Xen-virtualized builders) or the same machine where Launchpad (meaning "your launchpad.dev") is running, which is what this page details. Using the very same machine where Launchpad.dev is running limits the architectures you can build for: if you are running on i386, you can't build for PowerPC, for instace. * A builder needs to setup a bootstrap (what you get with debootstrap) Linux. That bootstrap is what you get in the chroot tarballs mentioned above. * When you upload a package and tell Launchpad to process it (see below for this), the builder will pick the chroot it needs (for instance, karmic i386), uncompress it and start building the package. * In summary, there are two lists: a list of builders and a list of chroots. |
You may have to calculate a new sha1sum of the chroot file. |
Line 88: | Line 75: |
* Run `scripts/process-upload.py /var/tmp/poppy` (creates hierarchy) | * Run `scripts/process-upload.py /var/tmp/txpkgupload` (creates hierarchy) |
Line 99: | Line 86: |
* `dput lpdev:~USERNAME/PPANAME/ubuntu some_source.changes` * `scripts/process-upload.py /var/tmp/poppy # Accept the source upload.` * Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete. * `scripts/process-accepted.py --ppa ubuntu # Create publishings for the binaries.` * `scripts/publish-distro.py --ppa # Publish the source and binaries.` |
* Find a source package `some_source` with a changes file `some_source.changes` * `dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes` * `scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.` * If this is your first time running soyuz locally, you'll also need to publish ubuntu: `scripts/publish-distro.py -C` * Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build"). * `scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.` * `scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.` * `scripts/publish-distro.py -vv --ppa # Publish the source and binaries.` |
Line 108: | Line 98: |
* Add yourself to `~ubuntu-team`. | |
Line 110: | Line 99: |
* `scripts/process-upload.py /var/tmp/poppy` | * `scripts/process-upload.py -vvv /var/tmp/txpkgupload` |
Line 114: | Line 103: |
* `scripts/process-accepted.py ubuntu` * `scripts/publish-distro.py` |
* `scripts/process-accepted.py -vv ubuntu` * `scripts/publish-distro.py -vv` |
You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side launchpad-buildd package on your system.
Initial setup
Run utilities/start-dev-soyuz.sh to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory.
Once you've set up your test database, run utilities/soyuz-sampledata-setup.py -e you@example.com (where you@example.com should be an email address you own and have a GPG key for). This prepares more suitable sample data in the launchpad_dev database, including recent Ubuntu series. If you get a "duplicate key" error, make schema and run again.
make run (or if you also want to use codehosting, make run_codehosting—some services may fail to start up because you already started them, but it shouldn't be a problem).
Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password test. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member ubuntu-team (conferring upload rights to the primary archive).
Extra PPA dependencies
The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want:
Log in as admin@canonical.com:test (I suggest using a different browser so you don't break up your ongoing session).
Open https://launchpad.dev/~ppa-user/+archive/test-ppa/+admin
- Edit external dependencies. They normally look like:
deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse
Setup a build slave
Installation
Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set lxc.aa_profile = unconfined in /var/lib/lxc/container-name/config which is required to disable AppArmor support.
If you are running Launchpad in a container, you will more than likely want your VM's network bridged on lxcbr0.
In your slave vm/lxc:
sudo apt-add-repository ppa:launchpad/buildd-staging
sudo apt-get update
sudo apt-get install launchpad-buildd bzr-builder quilt binfmt-support qemu-user-static
Alternatively, launchpad-buildd can be built from lp:launchpad-buildd with dpkg-buildpackage -b.
Edit /etc/launchpad-buildd/default and make sure ntphost points to an existing NTP server. You can check the NTP server pool to find one near you.
To run the build slave by default, you should make sure that other hosts on the Internet cannot send requests to it! Then:
echo RUN_NETWORK_REQUESTS_AS_ROOT=yes > ./etc/default/launchpad-buildd
Launchpad Configuration
From your host system:
Get an Ubuntu buildd chroot from Launchpad, using manage-chroot from lp:ubuntu-archive-tools:
manage-chroot -s precise -a i386 get
LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set
Point url to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)
Bob, shortly thereafter should report a successful status of 'idle'.
Drive slave through rpc
With librarian running, fire up a python shell and:
import xmlrpclib proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc') proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '') proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {}, {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'}) proxy.status() proxy.clean() # Clean up if it failed
You may have to calculate a new sha1sum of the chroot file.
Upload a source to the PPA
Run scripts/process-upload.py /var/tmp/txpkgupload (creates hierarchy)
Add to ~/.dput.cf:
[lpdev] fqdn = ppa.launchpad.dev:2121 method = ftp incoming = %(lpdev)s login = anonymous
Find a source package some_source with a changes file some_source.changes
dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes
scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.
If this is your first time running soyuz locally, you'll also need to publish ubuntu: scripts/publish-distro.py -C
- Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build").
scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.
scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.
scripts/publish-distro.py -vv --ppa # Publish the source and binaries.
- Note that private archive builds will not be dispatched until their source is published.
Dealing with the primary archive
dput lpdev:ubuntu some_source.changes
scripts/process-upload.py -vvv /var/tmp/txpkgupload
- Watch the output -- the upload might end up in NEW.
- If it does, go to the queue and accept it.
- Your builder should now be busy. Once it finishes, the binaries might go into NEW. Accept them if required.
scripts/process-accepted.py -vv ubuntu
scripts/publish-distro.py -vv
The first time, add -C to ensure a full publication of the archive.