#format irc
<Ursinha> #startmeeting
<Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
<Ursinha> [TOPIC] Roll Call
<Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
<MootBot> Meeting started at 10:02. The chair is Ursinha.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<MootBot> New Topic:  Roll Call
<bigjools> me
<Ursinha> me
<henninge> ich
<sinzui> me
<rockstar> me
* flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
<flacoste> me
<herb> me
<Ursinha> herb, intellectronica, hi
<Ursinha> :)
<Ursinha> who's missing?
<Ursinha> stub is missing, but he can join later
* mthaddon (n=mthaddon@adsl-70-137-141-190.dsl.snfc21.sbcglobal.net) has joined #launchpad-meeting
<Ursinha> intellectronica is missing too
<Ursinha> let's move on
<Ursinha> [TOPIC] Agenda
<Ursinha> * Actions from last meeting
<Ursinha> * Oops report & Critical Bugs
<Ursinha> * Operations report (mthaddon/herb/spm)
<Ursinha> * DBA report (stub)
<MootBot> New Topic:  Agenda
<Ursinha> [TOPIC] * Actions from last meeting
<Ursinha> * Ursinha to talk to intellectronica about bug 357316
<Ursinha> * Ursinha to talk to henninge about bug 302449
<Ursinha> * rockstar to confirm that bzr fix for bug 360791 was applied to LP's bzr tree.
<Ursinha> * cprov to request CP of fix for bug 370513
<MootBot> New Topic:  * Actions from last meeting
<ubottu> Launchpad bug 357316 in malone "hwdb +submit failing with KeyError OOPS" [Undecided,Triaged] https://launchpad.net/bugs/357316
<ubottu> Launchpad bug 302449 in rosetta "Uploading a file with the same name triggers a database constraint." [Medium,Triaged] https://launchpad.net/bugs/302449
<Ursinha> I suck and failed mine
<ubottu> Launchpad bug 360791 in bzr/1.14 "bzr pull/branch shows "Error received from smart server: ('NoSuchRevision',)"" [Critical,In progress] https://launchpad.net/bugs/360791
<ubottu> Launchpad bug 370513 in soyuz "failure to accept PPA uploads" [Critical,Fix committed] https://launchpad.net/bugs/370513
<Ursinha> [action] Ursinha to talk to intellectronica about bug 357316
<MootBot> ACTION received:  Ursinha to talk to intellectronica about bug 357316
<Ursinha> henninge, hi :)
<henninge> I think danilo was onto that ...
<rockstar> I don't know if the fix has been cherry picked into production...
<Ursinha> henninge, can you just confirm that, please? it's set as medium, do we use that status?
<henninge> Ursinha: in rosetta we do ;)
<Ursinha> rockstar, hm, can you check that too?
<rockstar> Code team does, for things of medium importance.
<bigjools> as does Soyuz
<herb> rockstar: it was cherry picked on 2009-05-09
<Ursinha> I rarely see medium statuses :) that's why I'm asking
<Ursinha> thanks herb
<rockstar> herb: cool, thanks.
<Ursinha> [action] henninge to check with danilo the status of bug 302449
<MootBot> ACTION received:  henninge to check with danilo the status of bug 302449
<ubottu> Launchpad bug 302449 in rosetta "Uploading a file with the same name triggers a database constraint." [Medium,Triaged] https://launchpad.net/bugs/302449
<Ursinha> cool
<Ursinha> moving on then
<Ursinha> [TOPIC] * Oops report & Critical Bugs
<Ursinha> there's only one worth mentioning, that is the one causing the InterfaceError oopses, we're still having lots and lots of occurrences (bugs 374909 and 376207), seems to be worked on by jamesh, is that correct flacoste?
<MootBot> New Topic:  * Oops report & Critical Bugs
<ubottu> Launchpad bug 374909 in storm "InterfaceError: connection already closed should be converted into DisconnectionError" [High,Triaged] https://launchpad.net/bugs/374909
<ubottu> Launchpad bug 376207 in launchpad-foundations "LaunchpadOpenIDStore doesn't support database disconnection" [High,In progress] https://launchpad.net/bugs/376207
<flacoste> so, jamesh is working on 374909
<flacoste> and the other one also
<Ursinha> right
<flacoste> but stuart has an easy fix for the later, that I'll likely asked to be cherrypicked
<flacoste> we can deploy jamesh proper fix during next roll-out
<Ursinha> flacoste, hm, good
<Ursinha> flacoste, about the cp, when do you think it can be done?
<flacoste> i didn't look at the branch
<herb> 374909 is still cropping up from time to time, btw. though it's much gooder(tm) than it was last week.
<flacoste> but once i approved it, as soon as the LOSA can take care of it
<Ursinha> flacoste, right. okay
<flacoste> i don't think i'll ask a C-P of the INterfaceError
<flacoste> (storm fix being worked on by jamesh)
<herb> flacoste: why?
<flacoste> well, because that's not a root-cause
<flacoste> so with the other fix in place, we shouldn't see it
<herb> ok
<flacoste> that fix is more prophylactic
<Ursinha> flacoste, who's investigating the root cause?
<flacoste> if you are talking about why we are getting disconnection errors in the first place
<flacoste> nobody, really
<flacoste> but we have a fix for the places where we should be trapping those
<flacoste> and recovering
<Ursinha> yes, the disconnection errors
<flacoste> we have no ideas at why it's happening
<flacoste> there is nothing in the DB logs
<flacoste> about them
<Ursinha> this is creepy
<Ursinha> the fix you say you have
<Ursinha> is inside the fixes for one of those bugs?
<flacoste> yes
<Ursinha> great
<Ursinha> so, you'll ask a cp for the second bug
<flacoste> exactly
<Ursinha> [action] flacoste to ask a cp for fix for bug 376207 after reviewing it
<MootBot> ACTION received:  flacoste to ask a cp for fix for bug 376207 after reviewing it
<ubottu> Launchpad bug 376207 in launchpad-foundations "LaunchpadOpenIDStore doesn't support database disconnection" [High,In progress] https://launchpad.net/bugs/376207
<Ursinha> awesome
<Ursinha> we have one critical bug, being worked on
<Ursinha> so, unless anyone has anything to point, moving to the next section!
<Ursinha> good
<Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> 2009-05-07 - Cherry pick r8906 to the scripts server and r122 of storm to lpnet* & edge*
<herb> 2009-05-09 - Cherry pick r4006 of bzr to the codehosting server and r123 of storm to lpnet* & edge*
<herb> 2009-05-09 - Cherry pick r8348, r8312 to the PPA server and r8376 to lpnet*
<herb> 2009-05-10 - mailman didn't have access the necessary access to the DB server, but it was only noticed after restarting for log rotation. mailing lists were unavailable for approximately 7 hours.
<herb> We still seem to be encountering bug #156453 and bug #118626, but the situation is much improved since the rollout.
<herb> flacoste: cprov requested a rollout of the current production tree to cesium. Apparently there was a critical fix that was included in while in RC, but we didn't re-roll to cesium. Can you (dis)approve that?
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<ubottu> Launchpad bug 118626 in bzr-email "plugin documentation does not make interaction with checkouts clear" [Medium,Confirmed] https://launchpad.net/bugs/118626
<bigjools> herb: it's already approved by kiko
<flacoste> herb: i think kiko did? but otherwise, i can look into it
<flacoste> right
<herb> bigjools: missed it.  thanks
<Ursinha> any other questions to herb?
<Ursinha> okay
<Ursinha> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<rockstar> herb: I was under the impression that the loggerhead stuff was WAY better than "much improved"
<flacoste> stub sent it to the list
<herb> rockstar: we're still restarting a couple of times a week. which is much improved over a couple of times a day.
<herb> rockstar: order of magnitude.
<Ursinha> flacoste, to lp list? I can't seem to find it
<rockstar> herb: okay.  Is it memory restarts, or hanging restarts
<flacoste> The ex-master database (launchpad_prod on hackberry) is bloated and
<flacoste> started exceeding its free space map settings. Nothing really to worry
<flacoste> about - it might cause bloat to spiral but I suspect not in this case.
<flacoste> The losas can bounce it after shutting down the systems using it as a
<flacoste> slave, and I've suggested using it as the standalone replica for
<flacoste> read-only mode launchpad during the rollout because we then rebuild it
<rockstar> (The memory restarts shouldn't be happening anymore)
<flacoste> afterwards and it will be all nice and freshly packed.
<flacoste> Nothing major do do with database patches this cycle. rockstar's
<flacoste> bugbranch and specbranch column pruning needs to be cleared with Mark
<flacoste> still.
<Ursinha> thanks flacoste 
<rockstar> flacoste: yeah, and there are other branches dependent on that one.
<herb> rockstar: The memory situation seems ok (ie. not death spiraling).  seems to be hangs at this point.
<herb> rockstar: we still see it ~1.5GB resident, but doesn't seem to grow beyond that.
<rockstar> herb: well, "death spiraling" to be is different than the memory issues.
<rockstar> herb: it's going to be *kinda* memory intensive just because of what it's serving.
<herb> rockstar: understood
<herb> rockstar: as I said, much improved. 1.2 - 1.5G is much better than 3.7G
<rockstar> herb: and we know the cause of the hangs, we just don't know how to fix it.
<herb> rockstar: good news, bad news, eh?
<rockstar> herb: yeah, something like that.
<Ursinha> okay. anyone else want to say something?
<Ursinha> 5
<Ursinha> 4
<Ursinha> 3
<Ursinha> 2
<Ursinha> 1
<Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
<Ursinha> #endmeeting