Host Ensemble Formula Development in Launchpad

Contact: Jonathan Lange
On Launchpad: none yet

Rationale

We would like as many formula as possible for Ensemble as quickly as possible. We can help do this by very quickly providing a top-notch developer collaboration experience for Ensemble using Launchpad. This will help the soon-to-be big Ensemble formula developer community.

Stakeholders

• Gustavo Niemeyer, Ensemble technical lead, last spoken 2011-03-04
• Robert Collins, Launchpad technical architect, last spoken 2011-03-07

• Clint Byrum, Ubuntu Server & formula developer, last spoken 2011-03-07

• James Troup, Canonical IS, last spoken 2011-03-04

Constraints and Requirements

Must

• Be able to develop this very quickly, for very low cost
• Get something up quickly that will promote Ensemble formula development

  • Store & share version-controlled source

    • Different versions of formula for different releases of the known-good set
  • Bug tracker that allows release management across a known-good set
  • Track and view bugs in a specific formula
  • Track bugs that affect multiple formulas
  • As a formula developer I want to associate formulas with an Ubuntu release, so that they can be properly indexed (known-good-set for release N) and also made dependent on the right base image
  • As a formula developer I want to push formulas to a personal (non-private) location so that other people can use the formula, and also fork a personal version of the formula for evolving it.
• Manage a release of a known-good set of formulas (sometimes called a "distribution")
• Search for formulas from the ensemble command-line client so that a user can find alternatives to the Ensemble KGS formulas that better fit their needs
  • As an ensemble user I want to search for formulas from the known-good-set for release N by default so that I can rapidly and confidently deploy services
  • As an ensemble user I want to search for formulas from all the universe of formulas including personal ones on request so that I am not constrained by Canonical's known-good-set
• Need to serve formulas without their version controlled history for fast downloads from the ensemble client

• Clients need to be able to pull & search for formula across multiple repositories so that people who work with private repositories available only with their LAN

  • Enable people to deploy formulas from the known-good-set for release N by default, and from a specific personal location on request.
• Handle a known-good set of roughly 2,000 formulas
• Handle a total set of around 100,000 formulas
  • multiple versions of the known-good set
  • unofficial versions of formulas
  • versions that are unpackaged
  • this number is basically made up
• Enable Ensemble to solve dependencies using information from published formulas.

Thoughts?

Implementation approaches

Create a distribution for Ensemble KGS, treat formulas as source packages

• At the moment, Launchpad doesn't allow bugs to be filed on a source package in a distribution unless that source package has been uploaded & built for that distribution

  • This prevents people filing bugs in Ubuntu for packages that are only in PPAs
  • Need to be careful when changing this constraint

• We would have to fix bug 386596

Create a project for Ensemble KGS, make formula branches series branches

Stick with Principia

Create a project group for Ensemble KGS, have a formula per project

Discussions

2011-03-07, #launchpad-dev, Freenode:

<SpamapS> lifeless: ping.. trying to wrap my head around adding a new distro to launchpad that is not for built packages...
<lifeless> SpamapS: hi
<SpamapS> lifeless: I understand you had a chat w/ jml about a distro for ensemble formulae
<lifeless> SpamapS: creating a distro is a privileged task; please wait for the ensemble list discussion and LEP before doing it
<SpamapS> lifeless: right, I'm not going to do it.. I am trying to understand the challenges.
<lifeless> SpamapS: ah, do you mean challenges in using or creating? :)
<SpamapS> lifeless: well I don't understand the suggestion that there's no bug tracking if you don't have a built package. How do non-built distros work given that restriction?
<lifeless> SpamapS: they don't
<lifeless> SpamapS: try and file a bug on openssh in baltix.
<SpamapS> Oh.
<lifeless> I dare you
<SpamapS> what's the use of distros then? ;)
* SpamapS is tempted now
<lifeless> SpamapS: distros are all about building and shipping packages
<SpamapS> Yeah, be that the case in reality.. in my head, they're about tracking collections of software. :-P
<lifeless> SpamapS: so we don't permit bugs filed on package names we know about unless the package is in the distro
<lifeless> SpamapS: this stops folk filing bugs on e.g. cassandra before its actually in Ubuntu
<SpamapS> that makes perfect sense. :)
<lifeless> and this matters because all our metadata driving policy - like packagesets etc - depends on source package / binary packages existing.
<SpamapS> So the real issue is there's no way, other than building, to create a package.
<lifeless> *I* don't know how deep the rabbit hole goes
<SpamapS> The way I saw it working was a mapping of a set of directories into source package names...
<lifeless> SpamapS: you can map source packages into source package names
<SpamapS> But I'm starting to wonder if, for the beginning of the project, we shouldn't just use a flat bzr and let people use tags to distinguish the formula name.
<lifeless> SpamapS: that would be a -lot- simpler.
<lifeless> SpamapS: jml and I had trouble inferring /requirements/ vs /things that using a distro would let you do/
<SpamapS> Yeah I think people got excited about reusing the distro code (myself included)
<lifeless> In some ways it makes a lot of sense
<lifeless> in other ways we really have two quite different things wedged into one in lp's distro bug support
<SpamapS> If the project takes off... we hope to have similar numbers of formulae as we have packages in Ubuntu so it would be very hard to track bugs on such a large body of work.
<lifeless> one things is 'lightweight components in a bug tracker' and the other is 'policy driven by uploads of stuff'
<lifeless> SpamapS: really? 20K forumulas?
<SpamapS> a formula for anything that opens a network port
<lifeless> SpamapS: thats more like 1K
<SpamapS> so, maybe 2000
<lifeless> we can start with one structure and migrate
<lifeless> like, it would be great to have lightweight components for e.g. bzr
<lifeless> something more than tags and less than separate projects
<SpamapS> Yeah.. I think thats probably going to be the thing I recommend in the list discussion.
<SpamapS> Tight coupling, once again, proves to make things less usable...
<lifeless> indeed
<lifeless> I'm a big fan of proving a core facility and letting folk do something cool with it
<lifeless> now, on the vcs side
<lifeless> jml said we need branch per (formula, release) tuple
<lifeless> which the (packagename, series) namespace in distro branches supplies
<lifeless> SpamapS: oh, on bugs, you /can/ use product series as a component like thing. same place in the UI
<lifeless> but the concept of nominations vs tasks would make it a little ugly
<SpamapS> lifeless: the series will be the series of formulae tho.. I'd like to keep that nice and clean.
<lifeless> indeed
<lifeless> so back to vcs
<lifeless> do you need that tuple now, or really just (formula, 'sid') - e.g. you only need trunks for the formulas?
<SpamapS> lifeless: I think having one bzr branch for all 2000 formulas would be a little painful. But you just mentioned in a recent discussion that bzr can branch the sub-dirs just fine, right?
<lifeless> SpamapS: you can work on just a subdir yes.
<lifeless> all the history comes, but honestly, 2K small projects is tiny bzr scale wise
<SpamapS> so if its  /principia/sid/formulas/databases/mysql  .. I can bzr branch lp:principia/sid/formulas/databases/mysql and interact with it, yes?
<SpamapS> Oh but I still get the whole history for /principia/sid .. :(
<SpamapS> A minor concession.. but one that our target market will hate.. because they're all git/ruby fanbois. ;)
<lifeless> SpamapS: yes, but I think you're overthinking the size here.
<lifeless> when you do this in git you get all the history too
<SpamapS> Right it's just faster. ;)
<SpamapS> which I *hate* as an argument for any vcs..
<SpamapS> but some people think it matters. ;)
<lifeless> SpamapS: mmm, not as such per various benchmarks we've done.
<lifeless> SpamapS: bzr network fetch is pretty good these days
<SpamapS> lifeless: Yeah, I'm going to again say that its a very small issue.. the main thing is to start getting those merge proposals cranking.. not achieve intergalactic domination overnight. :)
<lifeless> SpamapS: anyhow, i have not particular preference for what we do
<lifeless> I'm not trying to guide towards or away from distros
<lifeless> but - they are what they are.