Diff for "Soyuz/HowToUseSoyuzLocally"

Not logged in - Log In / Register

Differences between revisions 2 and 70 (spanning 68 versions)
Revision 2 as of 2009-11-05 12:33:24
Size: 4409
Editor: wgrant
Comment: Fix bad start-soyuz.sh URL.
Revision 70 as of 2015-09-24 11:04:42
Size: 5811
Editor: cjwatson
Comment: get build slave installation instructions a bit closer to production
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side {{{launchpad-buildd}}} package on your system.
Line 3: Line 5:
 * Alter the following sections of the configuration (`configs/development/launchpad-lazr.conf`) as shown:  * Run {{{utilities/start-dev-soyuz.sh}}} to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory.
 * Once you've set up your test database, run {{{utilities/soyuz-sampledata-setup.py -e you@example.com}}} (where ''you@example.com'' should be an email address you own and have a GPG key for). This prepares more suitable sample data in the {{{launchpad_dev}}} database, including recent Ubuntu series. If you get a "duplicate key" error, {{{make schema}}} and run again.
 * `make run` (or if you also want to use codehosting, `make run_codehosting`—some services may fail to start up because you already started them, but it shouldn't be a problem).
 * Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password ''test''. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member {{{ubuntu-team}}} (conferring upload rights to the primary archive).


== Extra PPA dependencies ==

The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want:

 * Log in as `admin@canonical.com:test` (I suggest using a different browser so you don't break up your ongoing session).
 * Open https://launchpad.dev/~ppa-user/+archive/test-ppa/+admin
 * Edit external dependencies. They normally look like:
{{{
deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse
}}}


= Setup a build slave =

== Installation ==

 * Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set `lxc.aa_profile = unconfined` in `/var/lib/lxc/container-name/config` which is required to disable App``Armor support.

If you are running Launchpad in a container, you will more than likely want your VM's network bridged on `lxcbr0`.

In your slave vm/lxc:

 * `sudo apt-add-repository ppa:launchpad/buildd-staging`
 * `sudo apt-get update`
 * `sudo apt-get install launchpad-buildd bzr-builder quilt binfmt-support qemu-user-static`

Alternatively, launchpad-buildd can be built from `lp:launchpad-buildd` with `dpkg-buildpackage -b`.

 * Edit `/etc/launchpad-buildd/default` and make sure ntphost points to an existing NTP server. You can check the [[http://www.pool.ntp.org/|NTP server pool]] to find one near you.

To run the build slave by default, you should make sure that other hosts on the Internet cannot send requests to it! Then:

 * `echo RUN_NETWORK_REQUESTS_AS_ROOT=yes > ./etc/default/launchpad-buildd`

== Launchpad Configuration ==

From your host system:

 * Get an Ubuntu buildd chroot from Launchpad, using `manage-chroot` from [[https://code.launchpad.net/+branch/ubuntu-archive-tools|lp:ubuntu-archive-tools]]:
 * `manage-chroot -s precise -a i386 get`
 * `LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set`

 * Point `url` to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)

Bob, shortly thereafter should report a successful status of 'idle'.

= Drive slave through rpc =

With librarian running, fire up a python shell and:
Line 6: Line 62:
[archivepublisher]
root: /var/tmp/archive
base_url: http://archive.launchpad.dev/ # <- this

[builddmaster]
root: /var/tmp/builddmaster/
uploader: scripts/process-upload.py -Mvv --context buildd # <- and this
import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc')
proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '')
proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {},
  {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'})
proxy.status()
proxy.clean() # Clean up if it failed
Line 15: Line 71:
 * In /etc/hosts, point `archive.launchpad.dev` at 127.0.0.88 like the rest.
 * Add to `/etc/apache2/site-available/local-launchpad`:

{{{
 <VirtualHost 127.0.0.88:80>
  ServerName archive.launchpad.dev
  LogLevel debug
  DocumentRoot /var/tmp/archive
  <Directory /var/tmp/archive/>
   Order Deny,Allow
   Deny from all
   Allow from 127.0.0.0/255.0.0.0
   Options Indexes
  </Directory>
 </VirtualHost>
}}}

 * Grab and run http://williamgrant.id.au/f/1/2009/make-ubuntu-sane.py, to set up some Ubuntu series that still exist, along with some state such as permissions.
 * Grab and run http://williamgrant.id.au/f/1/2009/start-soyuz.sh, which will fire up all Soyuz services. This will eat lots of RAM.

= Configure an account and PPA =

 * `make run`
 * Create an account for yourself (`utilities/make-lp-user USERNAME`)
 * Log in with `USERNAME@example.com:test`.
 * Add your OpenPGP key
  * `gpg --keyserver keyserver.launchpad.dev --send-keys DEADBEEF`
  * Add it to your Launchpad account, as normal.
  * Check email in `root`'s mailbox, and use it to activate the key.
 * Sign the code of conduct.
 * Create a PPA.

= Set up the PPA =

 * Log out.
 * Log in as `admin@example.com:test`.
 * Browse to the new PPA.
 * Click "Administer".
 * Ensure that the PPA is allowed to build on non-virtual builders.
 * Add an external dependency so you don't have to import the entire Ubuntu archive just for a PPA to work:

{{{
deb http://archive.ubuntu.com/ubuntu %(series) main restricted universe multiverse
}}}

= Configure a buildd =

 * `cd lib/canonical/buildd`
 * `dpkg-buildpackage -b`
 * `sudo dpkg -i ../launchpad-buildd_50_all.deb`
 * Make it work.
  * Add `$SUDO cp /etc/resolv.conf /etc/hosts "$HOME/build-$BUILDID/chroot-autobuild/etc` to the end of `/usr/share/launchpad-buildd/slavebin/mount-chroot`, so name resolution works.
  * Might need to add `--umask=022` to `twistd` args in `/etc/init.d/launchpad-buildd`, or `pkg-create-dbgsym` fails obscurely.
  * (>= Karmic) Might need to alter "`dpkg-source: extracting`" output grepping in sbuild to "`dpkg-source: info: extracting`"
 * Get an Ubuntu buildd chroot.
  * Look for the latest build of your favourite architecture and series at https://launchpad.net/ubuntu/+builds.
  * Find the chroot SHA1 in the third line of the log.
  * Search for it at http://launchpadlibrarian.net/search?digest=SOMESHA1
  * Grab one of the listed URLs.
   * http://launchpadlibrarian.net/34617356/chroot-ubuntu-lucid-i386.tar.bz2 is the current lucid i386 chroot at the time of writing.
 * `scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 add -f chroot-ubuntu-lucid-i386.tar.bz2`
 * Mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)
You may have to calculate a new sha1sum of the chroot file.
Line 80: Line 75:
 * Run `scripts/process-upload.py /var/tmp/poppy` (creates hierarchy)  * Run `scripts/process-upload.py /var/tmp/txpkgupload` (creates hierarchy) 
Line 91: Line 86:
 * `dput lpdev:~USERNAME/PPANAME/ubuntu some_source.changes`
 * `scripts/process-upload.py /var/tmp/poppy # Accept the source upload.`
 * Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete.
 * `scripts/process-accepted.py --ppa ubuntu # Create publishings for the binaries.`
 * `scripts/publish-distro.py --ppa # Publish the source and binaries.`
 * Find a source package `some_source` with a changes file `some_source.changes`
 * `dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes`
 * `scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.`
 * If this is your first time running soyuz locally, you'll also need to publish ubuntu: `scripts/publish-distro.py -C`
 * Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build").
 * `scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.`
 * `scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.`
 * `scripts/publish-distro.py -vv --ppa # Publish the source and binaries.`
Line 100: Line 98:
 * Add yourself to `~ubuntu-team`.
Line 102: Line 99:
 * `scripts/process-upload.py /var/tmp/poppy`  * `scripts/process-upload.py -vvv /var/tmp/txpkgupload`
Line 106: Line 103:
 * `scripts/process-accepted.py ubuntu`
 * `scripts/publish-distro.py`
 * `scripts/process-accepted.py -vv ubuntu`
 * `scripts/publish-distro.py -vv`

You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side launchpad-buildd package on your system.

Initial setup

  • Run utilities/start-dev-soyuz.sh to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory.

  • Once you've set up your test database, run utilities/soyuz-sampledata-setup.py -e you@example.com (where you@example.com should be an email address you own and have a GPG key for). This prepares more suitable sample data in the launchpad_dev database, including recent Ubuntu series. If you get a "duplicate key" error, make schema and run again.

  • make run (or if you also want to use codehosting, make run_codehosting—some services may fail to start up because you already started them, but it shouldn't be a problem).

  • Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password test. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member ubuntu-team (conferring upload rights to the primary archive).

Extra PPA dependencies

The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want:

deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse

Setup a build slave

Installation

  • Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set lxc.aa_profile = unconfined in /var/lib/lxc/container-name/config which is required to disable AppArmor support.

If you are running Launchpad in a container, you will more than likely want your VM's network bridged on lxcbr0.

In your slave vm/lxc:

  • sudo apt-add-repository ppa:launchpad/buildd-staging

  • sudo apt-get update

  • sudo apt-get install launchpad-buildd bzr-builder quilt binfmt-support qemu-user-static

Alternatively, launchpad-buildd can be built from lp:launchpad-buildd with dpkg-buildpackage -b.

  • Edit /etc/launchpad-buildd/default and make sure ntphost points to an existing NTP server. You can check the NTP server pool to find one near you.

To run the build slave by default, you should make sure that other hosts on the Internet cannot send requests to it! Then:

  • echo RUN_NETWORK_REQUESTS_AS_ROOT=yes > ./etc/default/launchpad-buildd

Launchpad Configuration

From your host system:

  • Get an Ubuntu buildd chroot from Launchpad, using manage-chroot from lp:ubuntu-archive-tools:

  • manage-chroot -s precise -a i386 get

  • LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set

  • Point url to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)

Bob, shortly thereafter should report a successful status of 'idle'.

Drive slave through rpc

With librarian running, fire up a python shell and:

import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc')
proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '')
proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {},
  {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'})
proxy.status()
proxy.clean() # Clean up if it failed

You may have to calculate a new sha1sum of the chroot file.

Upload a source to the PPA

  • Run scripts/process-upload.py /var/tmp/txpkgupload (creates hierarchy)

  • Add to ~/.dput.cf:

[lpdev]
fqdn = ppa.launchpad.dev:2121
method = ftp
incoming = %(lpdev)s
login = anonymous
  • Find a source package some_source with a changes file some_source.changes

  • dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes

  • scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.

  • If this is your first time running soyuz locally, you'll also need to publish ubuntu: scripts/publish-distro.py -C

  • Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build").
  • scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.

  • scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.

  • scripts/publish-distro.py -vv --ppa # Publish the source and binaries.

    • Note that private archive builds will not be dispatched until their source is published.

Dealing with the primary archive

  • dput lpdev:ubuntu some_source.changes

  • scripts/process-upload.py -vvv /var/tmp/txpkgupload

  • Watch the output -- the upload might end up in NEW.
    • If it does, go to the queue and accept it.
  • Your builder should now be busy. Once it finishes, the binaries might go into NEW. Accept them if required.
  • scripts/process-accepted.py -vv ubuntu

  • scripts/publish-distro.py -vv

    • The first time, add -C to ensure a full publication of the archive.

Soyuz/HowToUseSoyuzLocally (last edited 2022-12-10 08:09:22 by jugmac00)