1 <INTRO> IRC session "Getting started with Launchpad development"
   2 <INTRO> See https://wiki.ubuntu.com/MeetingLogs/devweek0909/HackSoyuz.
   3 <INTRO> Given 3 September 2009 by Celso Providelo (cprov), Michael Nelson (noodles775), and William Grant (wgrant), as part of Ubuntu Dev Week
   4 <INTRO> Take it away, cprov, noodles775, and wgrant!...
   5 <noodles775>    My name's Michael Nelson and I've been working on Launchpad and Soyuz for around 9 months now.
   6 <noodles775>    Here's an overview of what's coming up over the next 40mins or so:
   7 <noodles775>    1. Grill a new soyuz hacker with questions.
   8 <noodles775>    2. A guided tour through the Soyuz code-base
   9 <noodles775>    3. Setting up a Soyuz test scenari
  10 <noodles775>    o
  11 <noodles775>    So - up first is our chance to grill the latest Soyuz hacker: wgrant! Since the open-sourcing of Soyuz with Launchpad, wgrant has - in his own time - pushed 20 (!) launchpad branches.
  12 <noodles775>    7 of these are soyuz-related branches (afaics).
  13 <noodles775>    wgrant: wanna introduce yourself?
  14 <wgrant>        I didn't think I'd done quite that many, but OK!
  15 <wgrant>        So, I've been an Ubuntu developer for a few years now.
  16 <noodles775>    (that's how many I'd counted that have been pushed - not necessarily merged :) ).
  17 <wgrant>        And at some point became interested in the infrastructure behind it all.
  18 <noodles775>    Cool!
  19 <noodles775>    So this is our chance to find out how William got started working on Launchpad and Soyuz, what the issues were, and what he'd recommend to others who want to hack on Soyuz.
  20 <wgrant>        So when it was sneakily open-sourced a month ago, I jumped straight in.
  21 <wgrant>        There are lots of bits and pieces I'd like to see fixed or improved, so it was really great to see the source releaed.
  22 <wgrant>        Hacking Soyuz (and Launchpad in general) is probably going to be a little intimidating at first.
  23 <wgrant>        But Launchpad developers are very helpful to new contributors, so they can give you a lot of guidance if you get lost.
  24 <noodles775>    BTW everyone: please have think up some good questions that will help you get started and post them in #ubuntu-classroom-chat
  25 <wgrant>        To get oriented for development, I'd start off by setting up a local development environment (https://dev.launchpad.net/Getting).
  26 <wgrant>        Then have a look at how the codebase is organised. Maybe poke around in the model a bit with 'make harness'.
  27 <wgrant>        See how things work using the SoyuzTestPublisher, which I believe noodles775 will explain later.
  28 <wgrant>        Once you've found your way around a bit, identify a little bug or feature on which you want to work.
  29 <wgrant>        The next step is to ask a Launchpad developer (in #launchpad-dev) about it. They'll advise you whether you're attempting the impossible, or otherwise tell you where to start.
  30 <wgrant>        That bit is quite important, as it can stop you from hitting dead-ends or attempting something that's just too difficult for a first-time hacker (as quite a few things are).
  31 <noodles775>    cprov: have you (or anyone else) ever tried to explain what soyuz does by analogy?
  32 <wgrant>        An analogy... a good question. It's a pretty complex creature, so I'm not sure where to start.
  33 <noodles775>    I've sometimes tried to think of what soyuz does as a blogging engine... something familiar, with some similarities...
  34 <noodles775>    But there are quite a few differences too.
  35 <noodles775>    OK, any other questions for wgrant ?
  36 <cprov> well, Soyuz encompasses a lot of subcomponents that takes debian source packages as input and produces debian repositories, but there is a lot of details in the middle.
  37 <noodles775>    Which is a great lead-in to 2. A guided tour through the Soyuz code-base - take it away cprov :)
  38 <cprov> Hi, my name is Celso Providelo and I've been working on Soyuz for the last 5 years (!)
  39 <cprov> so, I would like to point you to some piece of documentation I've created to guide users to the Soyuz code base.
  40 <cprov> https://wiki.ubuntu.com/CelsoProvidelo/SoyuzInfrastructureOverview
  41 <cprov> it has a descent (but not pretty) diagram, it illustrate what I mean by 'lots of details in the middle' before.
  42 <cprov> Soyuz is in reality a set of integrated tools/components for 'controlling' software packages.
  43 <cprov> It starts with the 'upload server', an FTP daemon that receives source packages uploaded by users using `dput/dupload`.
  44 <cprov> Sources are them passed to the 'upload processor' which verifies their consistency (packaging metadata) and stores its information in the Launchpad database.
  45 <cprov> the publication of the source automatically creates a build request, which is dealt by the 'build dispatching' component.
  46 <cprov> it pass the source to a 'builder', an isolated environment for running `debuild`.
  47 <cprov> Binaries resulted from the build process come back to the upload processor and are checked before getting stored in Launchpad.
  48 <cprov> QUESTION: c_korn: cprov: those are the builders ? https://launchpad.net/builders
  49 <cprov> c_korn: exactly, those are the current build machines.
  50 <cprov> In the wiki page I've mentioned above, there are pointers to the corresponding modules for each part of the systems
  51 <cprov> knowing the topology described in that diagram and where to look in the codebase will help you to find out what need has to be changed.
  52 <cprov> That document is very short on details, but I expect them to be added as long as we get more community contributors. Don't hesitate in adding questions or suggestion directly to it.
  53 <cprov> noodles775: the stage is yours, I guess.
  54 <noodles775>    cprov: I had a few questions myself...
  55 <noodles775>    <noodles775> QUESTION: What's an example of a inconsistency that the upload processor will find and reject?
  56 <cprov> noodles775: basically anything that doesn't pass a `lintian` check.
  57 <noodles775>    ah ok, so it's really just to check for things that people should do themselves before uploading.
  58 <cprov> exactly
  59 <noodles775>    OK, on to: 3. Setting up a Soyuz test scenario
  60 <cprov> the upload processor also checks consistency against the packages previously uploaded
  61 <noodles775>    When hacking on Launchpad’s soyuz application – and creating tests to verify that your new functionality works, you’ll often need sources or binaries published in very specific scenarios.
  62 <noodles775>    We're going to use a special test feature - the SoyuzTestPublisher - to publish sources and binaries to a PPA in our development environment - and watch the status' update live in the browser.
  63 <noodles775>    The SoyuzTestPublisher – as the name suggests – was created for this exact reason (by cprov) :)
  64 <noodles775>    So for this hands-on - you don't need any previous LP development experience... but you do need a Launchpad development setup.
  65 <noodles775>    If you've set up the Launchpad development environment properly according to http://dev.launchpad.net/Getting, you should be able to run the following command:
  66 <noodles775>    $ rocketfuel-branch soyuz-test-scenario
  67 <noodles775>    While that's going - can I get an idea of how many (if any) people are following along?
  68 <noodles775>    Great! As long as there's at least one person, it's worth doing :)
  69 <noodles775>    When that's finished, change into the soyuz-test-scenario directory.
  70 <noodles775>    We will be watching the new publications at:
  71 <noodles775>    https://launchpad.dev/~cprov/+archive/ppa
  72 <noodles775>    This page updates the build status column every 60 seconds by default, so instead of tapping your fingers while you wait I'd recommend specifying an update interval of 5 seconds for the dynamic updates
  73 <noodles775>    as shown in the patch: http://pastebin.ubuntu.com/264831/
  74 <noodles775>    You can apply that patch to your current branch with:
  75 <noodles775>    $ wget http://pastebin.ubuntu.com/264831/plain/ -O update_every_five.diff
  76 <noodles775>    $ bzr patch update_every_five.diff
  77 <noodles775>    Just let me know if I go too fast...
  78 <noodles775>    With that patch applied, run 'make run' in your branch directory in one terminal and 'make harness' to get a python console in another.
  79 <noodles775>    Now, using the python console, we'll first just grab a sample-data user who has a PPA.
  80 <noodles775>    >>> cprov = getUtility(IPersonSet).getByName('cprov')
  81 <noodles775>    A few imports that we need
  82 <noodles775>    >>> from lp.soyuz.tests.test_publishing import SoyuzTestPublisher
  83 <noodles775>    >>> from lp.soyuz.interfaces.publishing import PackagePublishingStatus
  84 <noodles775>    and then we create our Soyuz test publisher instance:
  85 <noodles775>    >>> publisher = SoyuzTestPublisher()
  86 <noodles775>    Next, we just to ensure that the publisher has default distroseries etc. setup:
  87 <noodles775>    >>> publisher.prepareBreezyAutotest()
  88 <noodles775>    And now for the fun, we'll create a new published source package:
  89 <noodles775>    >>> testsrc = publisher.getPubSource(sourcename='testsrc', archive=cprov.archive, status=PackagePublishingStatus.PUBLISHED)
  90 <noodles775>    Finally, we'll create the missing builds for this new source package, and commit it all to the db:
  91 <noodles775>    >>> builds = testsrc.createMissingBuilds()
  92 <noodles775>    >>> import transaction;transaction.commit()
  93 <noodles775>    Now open a browser at https://launchpad.dev/~cprov/+archive/ppa (or re-load) and you'll see the new 'testsrc' package with its pending builds.
  94 <noodles775>    We'll now update the build manually watching the status update itself in the browser window.
  95 <noodles775>    >>> from lp.soyuz.interfaces.build import BuildStatus
  96 <noodles775>    >>> build = builds[0]
  97 <noodles775>    >>> build.buildstate = BuildStatus.BUILDING
  98 <noodles775>    Just watch your browser window without refreshing... after you commit the transaction, you'll see the build status for your package update within 5 seconds:
  99 <noodles775>    >>> transaction.commit()
 100 <noodles775>    Did it work?
 101 <noodles775>    Now we update it to fully-built:
 102 <noodles775>    >>> build.buildstate = BuildStatus.FULLYBUILT
 103 <noodles775>    >>> transaction.commit()
 104 <noodles775>    Now we've got a successful build, but its binary has not been published,
 105 <noodles775>    Mouse-over the build icon to see a description of the current state.
 106 <noodles775>    So we'll fake the successful publication of the binary with the SoyuzTestPublisher...
 107 <noodles775>    >>> binary_pkg_release = publisher.uploadBinaryForBuild(build, 'testbin')
 108 <noodles775>    >>> binary_pub = publisher.publishBinaryInArchive(binary_pkg_release, cprov.archive, status=PackagePublishingStatus.PUBLISHED)
 109 <noodles775>    Again, be ready to watch it update:
 110 <noodles775>    >>> transaction.commit()
 111 <noodles775>    There you go! A brief intro to the SoyuzTestPublisher for testing soyuz publications.
 112 <noodles775>    I've created a screencast and paste of the script at:
 113 <noodles775>    http://micknelson.wordpress.com/2009/09/04/testing-launchpad-soyuz-features/
 114 <noodles775>    So, that's all we had... does anyone have any questions?
 115 <noodles775>    I guess not :) Well, hope it was useful! Remember, if you've got any questions later, you can always ask them on #launchpad-dev.
 116 <noodles775>    jcastro: ?
 117 <noodles775>    <c_korn> some final words to end the Ubuntu Developer Week ?
 118 <jcastro>       not really
 119 <jcastro>       see you guys next cycle? :)
 120 <jcastro>       \o/
 121 <noodles775>    :)
 122 <jcastro>       we'll have an open week coming up soon so there will be more tutorials, etc.
 123 <jcastro>       please feel free to send your feedback to myself or daniel holbach