= Notes from UDS Maverick, Belgium, May 2010 =

Three big requirements generally from Soyuz:
 1. Support ARM
   * derived distros/archives
   * rebuilds
   * sftp uploads (in progress!)

 1. Software Centre
   * API support to control private PPAs (create/remove subscriptions/tokens etc.)
   * meta-data publishing for PPAs (consider how this works with private PPAs)
   * new pocket for "coolapps"

 1. Make buildd-manager more scalable and faster
   * Need to make build uploads asynchronous
   * Consider re-architecting buildd-manager, look at these for messaging systems:
     * Twisted implementation of AMQP: https://edge.launchpad.net/txamqp
     * Celery: a Python Distributed Task Queue: http://celeryproject.org/
   * Requirements from IS etc:
     * Elmo: Change /builders so the queue length/time shows archive type so we get a better idea of priority (mainly during rebuilds)
     * Elmo: Ensure branch builds are scored at ~1000
     * Mark: Need more knobs to dynamically change the queue priorities and weightings


== ARM ==
Can use the language of branches to talk about archives: the ubuntu-on-arm archive is like an integration branch for the various arm work, which will in part happen in further derived archives.

 * Want an overlay archive (where versions of packages override potentially higher versions in the parent archive)
 * Need control over which changes in the parent archive appear in the overlay
 * Similar to the debian -> ubuntu merge process
   * runs automatically some of the time, periods where we carefully review
   * default behaviour should be to stay in sync
   * should be possible to avoid upgrades for particular packages (blacklist)
 * Other archives may be derived from the derived archive itself (consider Ubuntu -> arm archive -> project-specific archive)
 * Will need packageset-based ACLs and other related features of them
 * Will need API tools such as "which packages are newer in archive A compared to archive B?"
 * tools for syncing between archives will be paramount, also need automatic superseding and merging
 * We will need to store more data on source packages about their build requirements ("neon" etc) and match this with builder "capabilities"
   * This sounds like a good use case to extend ProcessorFamily et al
 * a way of enforcing version number rules in a particular archive would be good

 * Rebuilds:
   * Binary rebuild NMUs are required, toolchain changes can happen that require recompilation and we don't want to re-upload the sources
   * Some rebuilds can be thrown away, some are permanent for binNMU

 * Image building:
   * Requires root so will need a virtual builder
   * New web UI required
   * Need to map image to/from the config that built it

== Software Centre ==
 * Make a new file in the pool where s/.deb/.changelog/ and the update manager will pick up a PPAs changelog data.
 * Meta-indexes for PPAs that list more data about apps in that PPA.
   * Need to think about how this will work with private PPAs since the archive is using basic http auth.
   * Maybe just protect pool/ with .htaccess.
 * We need a new pocket for "coolapps" that will be used to do new packages outside of the SRU process
   * Pocket build-deps will be the same as -updates
 * Private PPAs should be manageable completely from the API:
   * Add/remove subscribers/tokens.  This will need a celebrity for the software centre app itself maybe?
   * Inspect existing subscriptions/tokens.  (tokens is already done)


= RAW GOBBY NOTES FROM UDS =
== ARM ==
{{{
= arm-m-archive-features =

 (main ubuntu archive)
    /\
    | 
 (ubuntu-on-arm archive)
    /\
    |
 (various further archives)
 
Can use the language of branches to talk about archives: the ubuntu-on-arm
archive is like an integration branch for the various arm work, which will in 
part happen in further derived archives.

 * Want to maintain an overlay archive over the main archive
 * A bit like archive + ppa, but one sources.list entry and no pinning
   * why is this a requirement?
 * ways to specify which packages are 'part' of the archive
  * everything but ..., only ...
 * need to control over which changes from parent archive appear in our archive means "overlay" might not be the best term
 * similar to the debian -> ubuntu merge process
   * runs automatically some of the time, periods where we carefully review
   * default behaviour should be to stay in sync
   * should be possible to avoid upgrades for particular packages (blacklist)
  * will ubuntu on arm's milestones be synchronized with ubuntu's?
  * binary copy archives / source full archive
  * there will be derived archives of the ubuntu-on-arm archive
  * analagous to testing/unstable relationship?
    * quiet complex and we need to learn first if thats what we want
 * want to use same tool to maintain relationship between ubuntu and arm archives
   as between the arm archive and further-derived archives
   * unity work for example could have been done in a derived archive

 * do we start with ubuntu on arm being minimal?
   * as we may want to rebuild all binaries in the archive with a new toolchain,
     obviously not having to rebuild 16000 packages is a good thing
   * but subsetting the archive is hard beyond existing main/universe/multiverse split
   * package sets
   * if archive is incomplete, updates to ubuntu may introduce or lose a build dependency
   * something about resurrecting binaries
 * rebuild requirements relate to binNMUs
   * launchpad doesn't support binNUMs yet
   * for now we can say "don't expire binaries" for the next cycle
 * can one add new-to-ubuntu packages to ubuntu-on-arm (derivative) archive?
   * yes (well, requires policy thinking ... yes is definitly true for topic archives)
 * policy related to taking not-yet upstreamed patches
 * usual question of "how built into launchpad?"
   * a core api is "how many packages are newer in archive a than archive b"

what happens in the remote tool:
 * manual actions need to go into tools
 * automatic work goes into launchpad.
 * operations used by launchpad to get the automatic behaviour needs to be made available to the archive tool.


 * copying packages in launchpad is cheap
 * need:
   * to be able to find packages are eligible for syncing
   * need something to then every $N hours to do this syncing
   * can allow for launchpad and non-launchpad implementations of each building block

 * what about people who want to run their own archive

 * full archive vs overlay is about not having to follow your parent's timescale
   * also toolchain rebuilds

 * use cases:
   * debian->ubuntu
   * rebuild archives
   * start with these!

 * can we extend PPAs?
 * implementation-wise should we start with a special kind of ppa?
 * each archive needs a special flag in the version number to block auto-syncs (e.g. 'ubuntu' for the debian->ubuntu sync)
 * not chnaged packages have the same version that is in ubuntu
 * changed packages should not be synched

 * need secure upload queue (sftp)

= arm-m-derived-archive-rebuilds =

Two ways to do rebuilds:

  * Launchpad
  * Some way involving a cluster managed by a friendly debian guy

Problems:

 * rebuild tests compete with ppa builds
   * builder hardware, pretty good though
   * soyuz scaling issues
     * process-upload serialization (should be fixed soon)
 * would like to do this for ARM
   * cross compiling will not work
   * hardware is slow (OO.org takes ~2 days)
   * qemu on even very fast hardware is too slow
   * the qemu-with-a-cross-compiler underneath approach has legs, but isn't ready yet

Use cases for rebuilds:

 * verifying the tool chain works
   * lucid all builds with its own toolchain
   * how much does the new version of gcc break?
 * we want to rebuild the archive with a new tool chain
   * here we keep the results
   * can be because of optimizations, or a bad compiler flag in previous builds
   * needs something like a binNMU if the results go back into the same archive
   * important to be able to measure the differences

"scorched earth" rebuilds to do with build dependency loops -- session later in the week

The context here is doing rebuilds for Launchpad-managed archives.

Iterated rebuids are useful.

Some part of the process involves running a script on a DC machine that takes an
hour -- difficult to expose through Launchpad.

Diskless archives?

The variance of ARM architecture leads towards the requirement of assigning
builds to particular builders.  Builder pools are related, but existing spec
does not cover this.  In general, there is a need to store more data about
buildds.

Something about checking if a build uses swap, and general performance
monitoring of builds.

We could always build into a derived archive, and possibly copy back into the
source archives.

== ACTIONS ==

 * implement binNMUs in Launchpad
 * API exposure for copy-archive in Launchpad
 * implementing derived archives would help too

= arm-m-development-environment =
== Discussion ==

 * Goal: branch subset of ubuntu packages and manage changes
  * experiments - short/adhoc / long running
  * multiple parents: e.g. integrate goodies from multiple PPAs
  * hierarchy: common archive -> project archive -> project variant archive etc.
   * automatic superseding and merging

 * Release/Freezes/ACLs
   * have automatic merged release branch owned by releaes team during development
     period that gets set to manual mode during freezes.
   * at release another branch is auto created that is not changeable??

 * getting started: command line tool to branch some archive; by default it starts
   by copying the binaries;
 * managing changes: webtool that visualizes relationship to parent archive:
   * changes in downstream archive
  * changes in upstream archive (merge o matic'ish)
 * some changes like gcc would require move a binary copy archive to a source/rebuild
   everything mode? is that true? gcc might just have changed for a bug fix/crash etc.
   * tracking build dependencies may be relevant here

 * a way of enforcing version number rules in a particular archive would be good
}}}

== SOFTWARE CENTRE ==

{{{
= foundations-m-software-center-roadmap =

Ubuntu Software Center Roadmap
==============================

Features planned (10.10):
  - Selling software
    - Sell one thing is success, zero is non-success, two is no more of a success
    - Implies billing system working
    - Not an app that has serial number requirement
  - New applications show up in the software center ("open the floodgates")
    Session on that later: 
    https://blueprints.launchpad.net/ubuntu/+spec/desktop-maverick-opportunistic-apps-stable-release
  - Design for only a few apps a week getting in
    - Need both a social and technical process, including for unblessing PPAs
      - Risk is that a trusted application is updated *after* the review process
        and leads to bad bad badness
      - Also need to avoid "being Apple" and having too much bureaucracy around both new apps and updates
    - Eg: FOSS developer has a PPA, files a bug, subscribes a team, they review it, then add a line to a bzr-checked out file, commit it, and then software center gets it a day later (when the system does its daily check for updates?)
  - no live search of launchpad

11.04
  - Allow a donation feature (depending on how developer enables it for his app)
  - APIs for developers publishing software
  - Selling content for free apps (11.04)
  - Deal with apps that require a serial number (software center would need to issue it)

Design changes:
  - none needed for selling software
  - list of past purchases? (for 11.04)
  - move the list of featured apps to the home screen
  - move from webkit to GTK+
  - merge two panes into one
  - subcategories => debtag support in Launchpad
  - Drag an installed application from software center onto launcher
  - Drag an *uninstalled* application from software center onto launcher and
    install it

Non-design changes:
 - Software Center needs a major speedup
 - Regression testing, in particular to avoid performance regressions
 - apt-url integration
 - SOC student: integrate GDebI
 - Ratings and reviews (depending on server deployment)
 - Want: archive-based meta-data indexes from Launchpad

Switchboard repository:
 - repository that tracks application-which ppa that Software Center queries
 - for-pay software will have to be treated differently, since if naively 
   implemented, their archive index will not be visible
   - protect the pool, not the indices
 - much nicer than app-install data:
   - user gets updated data by apt-get update rather than install of app-install-data
   - data gets updated at package build time
   - easier to manage post-release than a package (reviews etc can live there)
   

== Other comments ==
 * will we need a policy in terms of what can be uploaded to that PPA
 * Should be up front about how we don't have a DRM system and users could just share downloaded .debs if they wanted
 
 
== Actions ==

[TODO] mpt, Design ???
}}}