Diff for "Soyuz/HowToUseSoyuzLocally"

Not logged in - Log In / Register

Differences between revisions 58 and 71 (spanning 13 versions)
Revision 58 as of 2015-01-21 11:38:03
Size: 4451
Editor: cjwatson
Comment: poppy → txpkgupload
Revision 71 as of 2015-09-24 11:07:50
Size: 6217
Editor: cjwatson
Comment: slave-only instructions
Deletions are marked like this. Additions are marked like this.
Line 23: Line 23:
= Configure a buildd = = Setup a build slave =
Line 25: Line 25:
This assumes a local install of the {{{launchpad-buildd}}} package. A less intrusive setup using chroot is under [[BuildFarm/TryOutBuildSlave|TryOutBuildSlave]]. == Installation ==
Line 27: Line 27:
 * `cd lib/canonical/buildd`
 * `debian/rules package`
 * `dpkg-buildpackage -b`
 * `sudo dpkg -i ../launchpad-buildd_*_all.deb`
 * `sudo apt-get -f install`
 * Make it work.
  * Edit `/etc/launchpad-buildd/default` and make sure `ntphost` points to an existing NTP server. You can check the [[http://www.pool.ntp.org/|NTP server pool]] to find one near you.
 * Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set `lxc.aa_profile = unconfined` in `/var/lib/lxc/container-name/config` which is required to disable App``Armor support.

If you are running Launchpad in a container, you will more than likely want your VM's network bridged on `lxcbr0`.

In your slave vm/lxc:

 * `sudo apt-add-repository ppa:launchpad/buildd-staging`
 * `sudo apt-get update`
 * `sudo apt-get install launchpad-buildd bzr-builder quilt binfmt-support qemu-user-static`

Alternatively, launchpad-buildd can be built from `lp:launchpad-buildd` with `dpkg-buildpackage -b`.

 * Edit `/etc/launchpad-buildd/default` and make sure ntphost points to an existing NTP server. You can check the [[http://www.pool.ntp.org/|NTP server pool]] to find one near you.

To run the build slave by default, you should make sure that other hosts on the Internet cannot send requests to it! Then:

 * `echo RUN_NETWORK_REQUESTS_AS_ROOT=yes > ./etc/default/launchpad-buildd`

== Launchpad Configuration ==

From your host system:
Line 35: Line 50:
 * `manage-chroot -s precise -a i386 get`
 * `LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set`

 * Point `url` to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)

Bob, shortly thereafter should report a successful status of 'idle'.

If you want to test just the build slave without a Launchpad instance, then, instead of using `manage-chroot -l dev set`, you can copy the chroot tarball to `/home/buildd/filecache-default/`; the base name of the file should be its sha1sum. You'll need to copy any other needed files (e.g. source packages) into the cache in the same way. You can then send XML-RPC instructions to the slave as below.

= Drive slave through rpc =

With librarian running, fire up a python shell and:
Line 36: Line 64:
manage-chroot -s precise -a i386 get
LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set
import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc')
proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '')
proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {},
  {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'})
proxy.status()
proxy.clean() # Clean up if it failed
Line 39: Line 72:
 * Mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)
You may have to calculate a new sha1sum of the chroot file.

You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side launchpad-buildd package on your system.

Initial setup

  • Run utilities/start-dev-soyuz.sh to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory.

  • Once you've set up your test database, run utilities/soyuz-sampledata-setup.py -e you@example.com (where you@example.com should be an email address you own and have a GPG key for). This prepares more suitable sample data in the launchpad_dev database, including recent Ubuntu series. If you get a "duplicate key" error, make schema and run again.

  • make run (or if you also want to use codehosting, make run_codehosting—some services may fail to start up because you already started them, but it shouldn't be a problem).

  • Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password test. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member ubuntu-team (conferring upload rights to the primary archive).

Extra PPA dependencies

The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want:

deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse

Setup a build slave

Installation

  • Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set lxc.aa_profile = unconfined in /var/lib/lxc/container-name/config which is required to disable AppArmor support.

If you are running Launchpad in a container, you will more than likely want your VM's network bridged on lxcbr0.

In your slave vm/lxc:

  • sudo apt-add-repository ppa:launchpad/buildd-staging

  • sudo apt-get update

  • sudo apt-get install launchpad-buildd bzr-builder quilt binfmt-support qemu-user-static

Alternatively, launchpad-buildd can be built from lp:launchpad-buildd with dpkg-buildpackage -b.

  • Edit /etc/launchpad-buildd/default and make sure ntphost points to an existing NTP server. You can check the NTP server pool to find one near you.

To run the build slave by default, you should make sure that other hosts on the Internet cannot send requests to it! Then:

  • echo RUN_NETWORK_REQUESTS_AS_ROOT=yes > ./etc/default/launchpad-buildd

Launchpad Configuration

From your host system:

  • Get an Ubuntu buildd chroot from Launchpad, using manage-chroot from lp:ubuntu-archive-tools:

  • manage-chroot -s precise -a i386 get

  • LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set

  • Point url to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)

Bob, shortly thereafter should report a successful status of 'idle'.

If you want to test just the build slave without a Launchpad instance, then, instead of using manage-chroot -l dev set, you can copy the chroot tarball to /home/buildd/filecache-default/; the base name of the file should be its sha1sum. You'll need to copy any other needed files (e.g. source packages) into the cache in the same way. You can then send XML-RPC instructions to the slave as below.

Drive slave through rpc

With librarian running, fire up a python shell and:

import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc')
proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '')
proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {},
  {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'})
proxy.status()
proxy.clean() # Clean up if it failed

You may have to calculate a new sha1sum of the chroot file.

Upload a source to the PPA

  • Run scripts/process-upload.py /var/tmp/txpkgupload (creates hierarchy)

  • Add to ~/.dput.cf:

[lpdev]
fqdn = ppa.launchpad.dev:2121
method = ftp
incoming = %(lpdev)s
login = anonymous
  • Find a source package some_source with a changes file some_source.changes

  • dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes

  • scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.

  • If this is your first time running soyuz locally, you'll also need to publish ubuntu: scripts/publish-distro.py -C

  • Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build").
  • scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.

  • scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.

  • scripts/publish-distro.py -vv --ppa # Publish the source and binaries.

    • Note that private archive builds will not be dispatched until their source is published.

Dealing with the primary archive

  • dput lpdev:ubuntu some_source.changes

  • scripts/process-upload.py -vvv /var/tmp/txpkgupload

  • Watch the output -- the upload might end up in NEW.
    • If it does, go to the queue and accept it.
  • Your builder should now be busy. Once it finishes, the binaries might go into NEW. Accept them if required.
  • scripts/process-accepted.py -vv ubuntu

  • scripts/publish-distro.py -vv

    • The first time, add -C to ensure a full publication of the archive.

Soyuz/HowToUseSoyuzLocally (last edited 2022-12-10 08:09:22 by jugmac00)