#format IRC

<Ursinha> #startmeeting
<Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<Ursinha> [TOPIC] Roll Call
<Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
<MootBot> Meeting started at 10:00. The chair is Ursinha.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<MootBot> New Topic:  Roll Call
<bigjools> me
<Ursinha> me :)
<flacoste> me
<sinzui> me
<henninge> me
<rockstar> me
<Ursinha> herb: hi
<rockstar> I wish someone not on the dev team would "me"
<Ursinha> rockstar: me too
<Ursinha> anyway :)
<intellectronica> rockstar, Ursinha: you can get that, but it requires /a lot/ of herding
<Ursinha> herb: are you there by chance? :)
<Ursinha> intellectronica: indeed
-!- stub [n=stub@canonical/launchpad/stub] has joined #launchpad-meeting
<Ursinha> so here is stub
<Ursinha> only herb missing
<herb> me
<herb> sorry
<Ursinha> :)
<Ursinha> hi herb!
<Ursinha> moving on :)
<herb> hi
<Ursinha> [TOPIC] Agenda * Actions from last meeting * Oops report & Critical Bugs  * Operations report (mthaddon/herb/spm) * DBA report (stub)
<MootBot> New Topic:  Agenda * Actions from last meeting * Oops report & Critical Bugs  * Operations report (mthaddon/herb/spm) * DBA report (stub)
<Ursinha> argh
<Ursinha> [TOPIC] * Actions from last meeting
<Ursinha> • sinzui to request CP for bugs 341085 and 341117
<Ursinha> • ursinha to talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766 - done
<Ursinha> • ursinha to debug 340058 and write the results back to the report - done
<Ursinha> • bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-)
<MootBot> New Topic:  * Actions from last meeting
<ubottu> Bug 341085 on http://launchpad.net/bugs/341085 is private
<ubottu> Launchpad bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Low,Triaged] https://launchpad.net/bugs/341766
<Ursinha> my items are done
<bigjools> done
<sinzui> Ursinha: Done
<Ursinha> awesome!
<Ursinha> next
<Ursinha> [TOPIC] * Oops report & Critical Bugs 
<Ursinha> intellectronica, https://bugs.edge.launchpad.net/malone/+bug/329908
<Ursinha> rockstar, as we previously talked, https://bugs.edge.launchpad.net/launchpad-bazaar/+bug/328302
<MootBot> New Topic:  * Oops report & Critical Bugs
<ubottu> Ubuntu bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New]
<ubottu> Error: This bug is private
<Ursinha> is it?
 *          Ursinha  looks again
 *         rockstar doesn't think that's the right bug...
<intellectronica> Ursinha: no news about this one yet :(
<rockstar> Ursinha, yeah, it's private because it has query info
<Ursinha> bug 328302
<ubottu> Bug 328302 on http://launchpad.net/bugs/328302 is private
<Ursinha> it's this one
<Ursinha> rockstar: I told you about that a few mins ago :)
<rockstar> Ursinha, yeah, and it looks like it's In Progress.
<Ursinha> intellectronica: that bug is happening like everyday
<Ursinha> rockstar: I brought it here because you didn't comment on that
<Ursinha> rockstar: at -code, that is
<rockstar> Ursinha, I have nothing to comment on.  Looks like thumper is working on it.
<Ursinha> rockstar: so as I presumed, you know as much about that as I do
<rockstar> Ursinha, unfortunately, due to illness and travelling, I've been a bit out of touch.
<intellectronica> Ursinha: shit. ok, i'll try to tackle it soon
<Ursinha> rockstar: no problem :) just brought to attention to see if you had some info that I (neither the bug report) would have
<Ursinha> intellectronica: thanks for that
<Ursinha> [action] intellectronica to make efforts to take a look at bug 329908
<MootBot> ACTION received:  intellectronica to make efforts to take a look at bug 329908
<ubottu> Launchpad bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New] https://launchpad.net/bugs/329908
<Ursinha> :)
<intellectronica> :)
<Ursinha> about the code bug, I'll catch up with thumper later, if the status doesn't change
<Ursinha> thanks intellectronica and rockstar 
<rockstar> Ursinha, he commented and said he had a fix for it.
<rockstar> Takes the query down to 3s instead of 17s.
<Ursinha> rockstar: I saw that.
<Ursinha> thanks
<Ursinha> moving on
<Ursinha> all critical bugs are fix committed, yay!
<Ursinha> that's all for me in oops/bugs section
<Ursinha> next
<Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> 2009-03-20 - CP r7866 and r7900 to production.
<herb> Sounding like a broken record, bug 156453 and bug 118625 continue to be a problem.
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
<herb> There have been some problems with staging updates this week. Seem to be mostly DB related and resolved now. There is a staging update running now.
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<herb> sinzui has a couple of cherry pick requests pending approval.
<herb> Other than that it's been a quiet week. Unless there are any questions, that's it from the LOSAs unless there are questions.
<rockstar> herb, we may actually have an answer for you next week!  :)
 *           sinzui will speak with kiko.
<herb> rockstar: that is exciting news
<Ursinha> sinzui: those are the ones of the actions from last meeting?
<sinzui> I'm not certain those items are worthy of a CP
<rockstar> herb, mwhudson and I are going to be hacking together on this soon.
<Ursinha> [action] sinzui to talk to kiko about pending cp requests
<MootBot> ACTION received:  sinzui to talk to kiko about pending cp requests
<sinzui> hence those bugs were high
-!- SteveA [n=steve@canonical/stevea] has joined #launchpad-meeting
<Ursinha> sinzui: hm, I see
<Ursinha> sinzui: are you going to sort out with kiko?
<sinzui> I'll get his opinion
<Ursinha> rockstar: that's cool
<Ursinha> right sinzui, thanks
<Ursinha> anything else for herb?
<Ursinha> that sounds a no :)
<Ursinha> thanks herb
<herb> thanks Ursinha
<Ursinha> moving on
<Ursinha> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> If you see 'database in recovery mode', it means part of PostgreSQL has crashed. We have seen it this week on staging and twice with buildbot. We don't know the trigger yet. PostgreSQL will recover, but our code might not. When it happened on staging, it looks like the staging appserver didn't reconnect fully and needed to be manually restarted.
<stub> If anyone has statement logging on and triggers the bug, I'l like your log files.
<stub> I'll abuse an ec2 instance tomorrow in an attempt to trigger the bug and try to get a pointer to the trigger.
<stub> A new PostgreSQL update came through today which contains some crash fixes. With luck, we will upgrade and the problem will never be seen again.
<stub> All the bugs I can find in the updated database maintenance tools have been fixed, so staging updates should work fine now (barring the PostgreSQL crash).
<stub> Some bigish DB changes are part of the next rollout. We are now maintaining two slaves so the maintenance tools needed to be updated to cope. We have also split our single replication set into two now as part of keeping the SSO service running during Launchpad outages. With luck, nobody will notice and things will continue to run smoothly.
<stub> I'm done
<Ursinha> hm.
<Ursinha> anyone has questions for stub?
<Ursinha> stub: that happened only on staging, right?
<stub> Crash on staging and twice on ec2 with buildbot
<Ursinha> this explain "some" weird oopses
<Ursinha> hm, thanks stub 
<Ursinha> anyone else wants to say something?
<Ursinha> 3
<Ursinha> 2
<Ursinha> 1
<Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
<Ursinha> #endmeeting
<MootBot> Meeting finished at 10:18.