Diff for "Soyuz/HowToUseSoyuzLocally"

Not logged in - Log In / Register

Differences between revisions 3 and 69 (spanning 66 versions)
Revision 3 as of 2009-11-05 12:36:16
Size: 4404
Editor: wgrant
Comment: Attach the scripts.
Revision 69 as of 2015-08-25 03:47:34
Size: 5543
Editor: blr
Comment:
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side {{{launchpad-buildd}}} package on your system.
Line 3: Line 5:
 * Alter the following sections of the configuration (`configs/development/launchpad-lazr.conf`) as shown:  * Run {{{utilities/start-dev-soyuz.sh}}} to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory.
 * Once you've set up your test database, run {{{utilities/soyuz-sampledata-setup.py -e you@example.com}}} (where ''you@example.com'' should be an email address you own and have a GPG key for). This prepares more suitable sample data in the {{{launchpad_dev}}} database, including recent Ubuntu series. If you get a "duplicate key" error, {{{make schema}}} and run again.
 * `make run` (or if you also want to use codehosting, `make run_codehosting`—some services may fail to start up because you already started them, but it shouldn't be a problem).
 * Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password ''test''. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member {{{ubuntu-team}}} (conferring upload rights to the primary archive).


== Extra PPA dependencies ==

The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want:

 * Log in as `admin@canonical.com:test` (I suggest using a different browser so you don't break up your ongoing session).
 * Open https://launchpad.dev/~ppa-user/+archive/test-ppa/+admin
 * Edit external dependencies. They normally look like:
{{{
deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse
}}}


= Setup a build slave =

== Installation ==

 * Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set `lxc.aa_profile = unconfined` in `/var/lib/lxc/container-name/config` which is required to disable App``Armor support.

If you are running Launchpad in a container, you will more than likely want your VM's network bridged on `lxcbr0`.

In your slave vm/lxc:

 * `sudo apt-add-repository ppa:launchpad`
 * `sudo apt-get update`
 * `sudo apt-get install launchpad-buildd`

Alternatively, launchpad-buildd can be built from `lp:launchpad-buildd` with `dpkg-buildpackage -b`.

 * Edit `/etc/launchpad-buildd/default` and make sure ntphost points to an existing NTP server. You can check the [[http://www.pool.ntp.org/|NTP server pool]] to find one near you.
 

== Launchpad Configuration ==

From your host system:

 * Get an Ubuntu buildd chroot from Launchpad, using `manage-chroot` from [[https://code.launchpad.net/+branch/ubuntu-archive-tools|lp:ubuntu-archive-tools]]:
 * `manage-chroot -s precise -a i386 get`
 * `LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set`

 * Point `url` to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)

Bob, shortly thereafter should report a successful status of 'idle'.

= Drive slave through rpc =

With librarian running, fire up a python shell and:
Line 6: Line 59:
[archivepublisher]
root: /var/tmp/archive
base_url: http://archive.launchpad.dev/ # <- this

[builddmaster]
root: /var/tmp/builddmaster/
uploader: scripts/process-upload.py -Mvv --context buildd # <- and this
import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc')
proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '')
proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {},
  {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'})
proxy.status()
proxy.clean() # Clean up if it failed
Line 15: Line 68:
 * In /etc/hosts, point `archive.launchpad.dev` at 127.0.0.88 like the rest.
 * Add to `/etc/apache2/site-available/local-launchpad`:

{{{
 <VirtualHost 127.0.0.88:80>
  ServerName archive.launchpad.dev
  LogLevel debug
  DocumentRoot /var/tmp/archive
  <Directory /var/tmp/archive/>
   Order Deny,Allow
   Deny from all
   Allow from 127.0.0.0/255.0.0.0
   Options Indexes
  </Directory>
 </VirtualHost>
}}}

 * Grab and run [[attachment:make-ubuntu-sane.py|make-ubuntu-sane.py]], to set up some Ubuntu series that still exist, along with some state such as permissions.
 * Grab and run [[attachment:start-soyuz.sh|start-soyuz.sh]], which will fire up all Soyuz services. This will eat lots of RAM.

= Configure an account and PPA =

 * `make run`
 * Create an account for yourself (`utilities/make-lp-user USERNAME`)
 * Log in with `USERNAME@example.com:test`.
 * Add your OpenPGP key
  * `gpg --keyserver keyserver.launchpad.dev --send-keys DEADBEEF`
  * Add it to your Launchpad account, as normal.
  * Check email in `root`'s mailbox, and use it to activate the key.
 * Sign the code of conduct.
 * Create a PPA.

= Set up the PPA =

 * Log out.
 * Log in as `admin@example.com:test`.
 * Browse to the new PPA.
 * Click "Administer".
 * Ensure that the PPA is allowed to build on non-virtual builders.
 * Add an external dependency so you don't have to import the entire Ubuntu archive just for a PPA to work:

{{{
deb http://archive.ubuntu.com/ubuntu %(series) main restricted universe multiverse
}}}

= Configure a buildd =

 * `cd lib/canonical/buildd`
 * `dpkg-buildpackage -b`
 * `sudo dpkg -i ../launchpad-buildd_50_all.deb`
 * Make it work.
  * Add `$SUDO cp /etc/resolv.conf /etc/hosts "$HOME/build-$BUILDID/chroot-autobuild/etc` to the end of `/usr/share/launchpad-buildd/slavebin/mount-chroot`, so name resolution works.
  * Might need to add `--umask=022` to `twistd` args in `/etc/init.d/launchpad-buildd`, or `pkg-create-dbgsym` fails obscurely.
  * (>= Karmic) Might need to alter "`dpkg-source: extracting`" output grepping in sbuild to "`dpkg-source: info: extracting`"
 * Get an Ubuntu buildd chroot.
  * Look for the latest build of your favourite architecture and series at https://launchpad.net/ubuntu/+builds.
  * Find the chroot SHA1 in the third line of the log.
  * Search for it at http://launchpadlibrarian.net/search?digest=SOMESHA1
  * Grab one of the listed URLs.
   * http://launchpadlibrarian.net/34617356/chroot-ubuntu-lucid-i386.tar.bz2 is the current lucid i386 chroot at the time of writing.
 * `scripts/ftpmaster-tools/manage-chroot.py -s lucid -a i386 add -f chroot-ubuntu-lucid-i386.tar.bz2`
 * Mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)
You may have to calculate a new sha1sum of the chroot file.
Line 80: Line 72:
 * Run `scripts/process-upload.py /var/tmp/poppy` (creates hierarchy)  * Run `scripts/process-upload.py /var/tmp/txpkgupload` (creates hierarchy) 
Line 91: Line 83:
 * `dput lpdev:~USERNAME/PPANAME/ubuntu some_source.changes`
 * `scripts/process-upload.py /var/tmp/poppy # Accept the source upload.`
 * Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete.
 * `scripts/process-accepted.py --ppa ubuntu # Create publishings for the binaries.`
 * `scripts/publish-distro.py --ppa # Publish the source and binaries.`
 * Find a source package `some_source` with a changes file `some_source.changes`
 * `dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes`
 * `scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.`
 * If this is your first time running soyuz locally, you'll also need to publish ubuntu: `scripts/publish-distro.py -C`
 * Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build").
 * `scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.`
 * `scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.`
 * `scripts/publish-distro.py -vv --ppa # Publish the source and binaries.`
Line 100: Line 95:
 * Add yourself to `~ubuntu-team`.
Line 102: Line 96:
 * `scripts/process-upload.py /var/tmp/poppy`  * `scripts/process-upload.py -vvv /var/tmp/txpkgupload`
Line 106: Line 100:
 * `scripts/process-accepted.py ubuntu`
 * `scripts/publish-distro.py`
 * `scripts/process-accepted.py -vv ubuntu`
 * `scripts/publish-distro.py -vv`

You're going to run Soyuz in a branch you create for the purpose. To get the whole experience, you'll also be installing the slave-side launchpad-buildd package on your system.

Initial setup

  • Run utilities/start-dev-soyuz.sh to ensure that some Soyuz-related services are running. Some of these may already be running, in which case you'll get some failures that are probably harmless. Note: these services eat lots of memory.

  • Once you've set up your test database, run utilities/soyuz-sampledata-setup.py -e you@example.com (where you@example.com should be an email address you own and have a GPG key for). This prepares more suitable sample data in the launchpad_dev database, including recent Ubuntu series. If you get a "duplicate key" error, make schema and run again.

  • make run (or if you also want to use codehosting, make run_codehosting—some services may fail to start up because you already started them, but it shouldn't be a problem).

  • Open https://launchpad.dev/~ppa-user/+archive/test-ppa in a browser to get to your pre-made testing PPA. Log in with your own email adddress and password test. This user has your GPG key associated, has signed the Ubuntu Code of Conduct, and is a member ubuntu-team (conferring upload rights to the primary archive).

Extra PPA dependencies

The testing PPA has an external dependency on Lucid. If that's not enough, or not what you want:

deb http://archive.ubuntu.com/ubuntu %(series)s main restricted universe multiverse

Setup a build slave

Installation

  • Create a new trusty virtual-machine with kvm (recommended), or alternatively a trusty lxc container. If using lxc, set lxc.aa_profile = unconfined in /var/lib/lxc/container-name/config which is required to disable AppArmor support.

If you are running Launchpad in a container, you will more than likely want your VM's network bridged on lxcbr0.

In your slave vm/lxc:

  • sudo apt-add-repository ppa:launchpad

  • sudo apt-get update

  • sudo apt-get install launchpad-buildd

Alternatively, launchpad-buildd can be built from lp:launchpad-buildd with dpkg-buildpackage -b.

  • Edit /etc/launchpad-buildd/default and make sure ntphost points to an existing NTP server. You can check the NTP server pool to find one near you.

Launchpad Configuration

From your host system:

  • Get an Ubuntu buildd chroot from Launchpad, using manage-chroot from lp:ubuntu-archive-tools:

  • manage-chroot -s precise -a i386 get

  • LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 manage-chroot -l dev -s precise -a i386 -f chroot-ubuntu-precise-i386.tar.bz2 set

  • Point url to your slave's IP and mark Bob the Builder as OK (https://launchpad.dev/builders/bob/+edit)

Bob, shortly thereafter should report a successful status of 'idle'.

Drive slave through rpc

With librarian running, fire up a python shell and:

import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://localhost:8221/rpc')
proxy.ensurepresent('d267a7b39544795f0e98d00c3cf7862045311464', 'http://launchpad.dev:58080/93/chroot-ubuntu-lucid-i386.tar.bz2', '', '')
proxy.build('1-1', 'translation-templates', 'd267a7b39544795f0e98d00c3cf7862045311464', {},
  {'archives': ['deb http://archive.ubuntu.com/ubuntu/ lucid main'], 'branch_url': '/home/buildd/gimp-2.6.8'})
proxy.status()
proxy.clean() # Clean up if it failed

You may have to calculate a new sha1sum of the chroot file.

Upload a source to the PPA

  • Run scripts/process-upload.py /var/tmp/txpkgupload (creates hierarchy)

  • Add to ~/.dput.cf:

[lpdev]
fqdn = ppa.launchpad.dev:2121
method = ftp
incoming = %(lpdev)s
login = anonymous
  • Find a source package some_source with a changes file some_source.changes

  • dput -u lpdev:~ppa-user/test-ppa/ubuntu some_source.changes

  • scripts/process-upload.py /var/tmp/txpkgupload -C absolutely-anything -vvv # Accept the source upload.

  • If this is your first time running soyuz locally, you'll also need to publish ubuntu: scripts/publish-distro.py -C

  • Within five seconds of upload acceptance, the buildd should start building. Wait until it is complete (the build page will say "Uploading build").
  • scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster # Process the build upload.

  • scripts/process-accepted.py -vv --ppa ubuntu # Create publishings for the binaries.

  • scripts/publish-distro.py -vv --ppa # Publish the source and binaries.

    • Note that private archive builds will not be dispatched until their source is published.

Dealing with the primary archive

  • dput lpdev:ubuntu some_source.changes

  • scripts/process-upload.py -vvv /var/tmp/txpkgupload

  • Watch the output -- the upload might end up in NEW.
    • If it does, go to the queue and accept it.
  • Your builder should now be busy. Once it finishes, the binaries might go into NEW. Accept them if required.
  • scripts/process-accepted.py -vv ubuntu

  • scripts/publish-distro.py -vv

    • The first time, add -C to ensure a full publication of the archive.

Soyuz/HowToUseSoyuzLocally (last edited 2022-12-10 08:09:22 by jugmac00)