## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 11:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<sinzui> me
<matsubara> Chex, gary_poster, bigjools, allenap, rockstar, danilos: Hi
<gary_poster> me
<bigjools> me sort of
<danilos> me
<Chex> me
<allenap> me
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * QA stats of the week
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * gary_poster and sinzui to discuss about the issues between master and slave that could be causing oopses
<matsubara>  * allenap to follow up with deryck about bug 230106
<matsubara>  * gary_poster to ask stub to investigate bug 556245
<matsubara>  * Ursinha to send an email to stub asking for the dba report
<matsubara>    * Done
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<sinzui> matsubara, done
<matsubara> thanks sinzui 
<gary_poster> matsubara: we have progress, in that meliae is installed thanks to salgado.  We're still at the diagnosis stage.
<matsubara> thanks gary_poster 
<gary_poster> we need to get the change CP'd.
<matsubara> hwo about 556245?
<allenap> matsubara: I suck, I'm sorry I didn't follow that up except to comment on the bug.
<matsubara> allenap, any news about 230106
<matsubara> ?
<allenap> matsubara: See suckage above ^.
<matsubara> hmm ok. shall I keep the action item?
<allenap> matsubara: Yes please.
<matsubara> [action] * allenap to follow up with deryck about bug 230106
<MootBot> ACTION received:  * allenap to follow up with deryck about bug 230106
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> I have two oops bugs: https://bugs.edge.launchpad.net/launchpad-registry/+bug/568459 which sinzui already duped (Thanks dude!)
<ubottu> Ubuntu bug 568459 in launchpad-registry "LocationError raised browsing code of conduct page (dup-of: 555734)" [Undecided,New]
<ubottu> Ubuntu bug 555734 in launchpad-registry "oops signing CoC if keys mutate" [High,Fix committed]
<matsubara> gary_poster, and this one for foundations: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/568456 
<ubottu> Ubuntu bug 568456 in launchpad-foundations "GpgmeError raised importing public gpg key" [Undecided,New]
<gary_poster> ack, matsubara thanks
<matsubara> thanks gary_poster 
<sinzui> Is this fixed in our tree now
* sinzui does not see how lucid issue could be affecting hardy production
<sinzui> but gpgme is a lucid/lp issue that I believe was fixed by salgado
<matsubara> we had a couple of script failures, which spm already took care of
<matsubara> we have no critical bugs open
<matsubara> [action] gary_poster to triage bug 568456 
<MootBot> ACTION received:  gary_poster to triage bug 568456
<ubottu> Launchpad bug 568456 in launchpad-foundations "GpgmeError raised importing public gpg key" [Undecided,New] https://launchpad.net/bugs/568456
<matsubara> let's move on, thanks everyone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Chex> hi everyone
<Chex> here is the LOSA report for this week, just reporting on incidents this week:
<Chex> - LP incidents of note:
<Chex>         : 18-Apr:  swap problems on lpnet servers.  restarts on: lpnets 1,2,11,12,13,14,15 services; Bug: 531071
<ubottu> Launchpad bug 531071 in launchpad-foundations "lpnet app servers are leaking memory again" [High,Triaged] https://launchpad.net/bugs/531071
<Chex>         : 20-Apr: Checkwatches had hung, killed. Bug: 567088
<ubottu> Launchpad bug 567088 in malone "checkwatches keeps hanging at irregular intervals" [Undecided,New] https://launchpad.net/bugs/567088
<Chex>         : 21-Apr: crowberry h/w error, restarted the machine. OK for now, following up on the hardware issues.
<Chex>         : 21-Apr: mizuho apparently OOMK'd, rebooted; necessitating full LP restart; Bug 556245
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<Chex> does anyone have any questions or comments on these?
<gary_poster> Chex, are the LOSAs familiar with the steps necessary to use the meliae hooks?  I know salgado has been trying to communicate those via the right channels.
<gary_poster> (this is pertinent to the memory leaks in librarian and LP)
<matsubara> allenap, can you bring the checkwatches issue with your team?
<matsubara> allenap, nm, I just noticed that you commented on the bug already, thanks! :-)
<Chex> gary_poster: I was working with maliae breifly with the U1 team earlier this week, but I am unaware of the process for LP..
<gary_poster> Chex, could you get back to me or salgado with the best way to communicate these changes to the LOSAs?  I know he has updated the wiki pages with instructions
<gary_poster> (the LOSA wiki pages)
<Chex> gary_poster: sorry, I see it now, thru his email to the LOSA list, I will look at that and make sure the rest of the team is aware of that, also.
<gary_poster> Chex, thank you very much
<matsubara> [action] Chex to work with gary_poster or salgado to sort out how to use the  meliae hooks to debug librarian and LP memory leaks
<MootBot> ACTION received:  Chex to work with gary_poster or salgado to sort out how to use the  meliae hooks to debug librarian and LP memory leaks
<Chex> gary_poster: that is the correct way to communicate those, but hilighting these issues to us in these meetings helps, too.
<gary_poster> cool :-)
<matsubara> thanks Chex and gary_poster 
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> I'll ask stub to send the report to the list
<matsubara> [action] matsubara to ask stub for the DBA report
<MootBot> ACTION received:  matsubara to ask stub for the DBA report
<matsubara> [TOPIC] * QA stats of the week
<MootBot> New Topic:  * QA stats of the week
<matsubara> I don't have stats for this week. :-(
<matsubara> sorry, I'll add them later to the MeetingAgenda page. need to find the script which generates it
<matsubara> [action] matsubara to update qa stats in the MeetingAgenda page
<MootBot> ACTION received:  matsubara to update qa stats in the MeetingAgenda page
* gary_poster is happy because they would show shameful stats :-)  ...working on them now...
<matsubara> hehe
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<sinzui> gary_poster, I think everyone will see soyuz first
<matsubara> no new proposed items
<gary_poster> ok
<matsubara> sinzui, soyuz cleared their untriaged queue last week
* sinzui is shocked
<matsubara> anything else before I close?
<bigjools> feh
<matsubara> :-)
<matsubara> worst offenders are now foundations: launchpadlib and oops-tools
<matsubara> actually no, oops-tools has 0 new untriaged. I cleared that a few weeks ago
<matsubara> well, I guess that's it for today
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda  for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 11:19.