#format IRC
<Ursinha> #startmeeting
<Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<danilos> koo
<MootBot> Meeting started at 11:00. The chair is Ursinha.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<Ursinha> [TOPIC] Roll Call
<Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<MootBot> New Topic:  Roll Call
* rockstar  
<Ursinha> me me
<EdwinGrubbs> me
* salgado is now known as salgado-lunch
<matsubara> me
<Ursinha> matsubara, Chex, bigjools, hello
<Ursinha> :)
<danilos> me
<allenap> me
* Ursinha throws a shoe on bigjools 
* bigjools dodges the shoe and returns fire with a custard pie
<Chex> me
<Ursinha> so we're all here hehe
<Ursinha> [TOPIC] Agenda
<Ursinha>  * Actions from last meeting
<Ursinha>  * Oops report & Critical Bugs & Broken scripts
<Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
<Ursinha>  * DBA report (stub)
<Ursinha>  * QA stats of the week
<Ursinha>  * Proposed items
<MootBot> New Topic:  Agenda
<Ursinha> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<Ursinha> * allenap to follow up with deryck about bug 230106
<Ursinha> * gary_poster to triage bug 568456
<Ursinha> * Chex to work with gary_poster or salgado to sort out how to use the meliae hooks to debug librarian and LP memory leaks
<Ursinha> * matsubara to ask stub for the DBA report
<Ursinha>     * Done.
<Ursinha> * matsubara to update qa stats in the MeetingAgenda page
<Ursinha>     * Done.
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<ubottu> Launchpad bug 568456 in launchpad-foundations "GpgmeError raised importing public gpg key" [Low,Triaged] https://launchpad.net/bugs/568456
<Ursinha> bug 568456 is triaged, so done
<Ursinha> I'll be talking about bug 230106 in the oops section
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<Ursinha> so, moving on
<Ursinha> ah, not that fast, Chex, was your item sorted out?
<Ursinha> Chex, hello :)
<Chex> Ursinha: yes It was.. I will be notifying the losa's about the
<Chex> proper procedure for crash-dump generation on the lp servers today
<Ursinha> thanks Chex!
<Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Chex> I touched base with salgado today
<Ursinha> the most offenders are old ones
<Ursinha> still a lot of oopses daily due to bug 553361
<Ursinha> matsubara, do you know if that is going to be fixed in time for the rollout? 
<Ursinha> this is really annoying and it's "tainting" oops summaries :/
<ubottu> Launchpad bug 553361 in launchpad-foundations "OOPS when accessing launchpad with no referrer, eg. via wget" [High,Triaged] https://launchpad.net/bugs/553361
<matsubara> Ursinha, it will not
 mars matsubara maxb mbarnett MootBot mrevell mthaddon mwhudson 
 mars matsubara maxb mbarnett MootBot mrevell mthaddon mwhudson 
<Ursinha> matsubara, :(
<Ursinha> matsubara, do you know if that can be fixed for a CP later then?
<matsubara> Ursinha, always
<Ursinha> matsubara, not much later :)
<matsubara> I'll bring it up to gary on today's call
<Ursinha> thanks a lot matsubara 
<Ursinha> [action] matsubara to talk with gary_poster about  a CP for bug 553361
<MootBot> ACTION received:  matsubara to talk with gary_poster about  a CP for bug 553361
<ubottu> Launchpad bug 553361 in launchpad-foundations "OOPS when accessing launchpad with no referrer, eg. via wget" [High,Triaged] https://launchpad.net/bugs/553361
<Ursinha> also, we're having about 270 oopses of the "None: None" kind (OOPS-1579CEMAIL1) on a daily basis
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1579CEMAIL1
<Ursinha> we cannot say what's the problem because the oops report is almost empty, and we'd need to fix bug 230106 to have more info
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<Ursinha> deryck said it won't be fixed soon, but I guess that given the number of oopses maybe we should try something to diagnose that problem
<Ursinha> allenap, any suggestions?
<allenap> Ursinha: I'll bump the bug up to High and explain why. It is scheduled to happen soon, this will make it happen sooner.
<Ursinha> or anyone else
<Ursinha> thanks a lot allenap 
<matsubara> Ursinha, of course we know what's the problem
<matsubara> it's bug 54005, isn't it?
<ubottu> Launchpad bug 54005 in launchpad-foundations "bugs.launchpad.ubuntu.com email ending up in Launchpad mailbox" [Low,Triaged] https://launchpad.net/bugs/54005
<Ursinha> matsubara, no, we're not sure because the bug is too old
<matsubara> the fact they show up a None None is 230106
<Ursinha> yes, that's what I said :) we need to fix 230106 in order to find out if the 270 oopses are really 54005
<Ursinha> [action] allenap to bump bug 230106 to High and explain why
<MootBot> ACTION received:  allenap to bump bug 230106 to High and explain why
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<matsubara> Ursinha, we don't need bug 230106 fixed to fix bug 54005
<ubottu> Launchpad bug 54005 in launchpad-foundations "bugs.launchpad.ubuntu.com email ending up in Launchpad mailbox" [Low,Triaged] https://launchpad.net/bugs/54005
<Ursinha> matsubara, read what I said again :)
<matsubara> it's either 54005 is invalid because it's too old or we re-purpose it to fix OOPS-1579CEMAIL1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1579CEMAIL1
<Ursinha> we're having a higher number of timeouts as well, about 800, and that doesn't seem to be specific to a part of launchpad
<matsubara> I'm disagreeing that 230106 needs to fixed first
<Ursinha> matsubara, I didn't say that we need to fix one to fix the other, I said we need to fix one to be able to see if the oopses are really the other
<Ursinha> does anyone have a clue about the higher number of timeouts?
<matsubara> and that's what I'm disagreeing about
<Ursinha> matsubara, are you sure that oops is 54005?
<Ursinha> not all oopses None None are bugs email
<matsubara> pretty sure, but as I said, one doesn't need to be sure to fix OOPS-1579CEMAIL1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1579CEMAIL1
<matsubara> yes, some are from the spec tracker
<matsubara> which is also mentioned in the bug report
<Ursinha> matsubara, we discussed that two meetings ago, I guess, that's where this conclusion came from
<Ursinha> matsubara, but of course, you can discuss that bug with gary as well, that would be much appreciated :)
<matsubara> ok
<Ursinha> [action] matsubara to discuss bug 54005 with gary_poster
<MootBot> ACTION received:  matsubara to discuss bug 54005 with gary_poster
<ubottu> Launchpad bug 54005 in launchpad-foundations "bugs.launchpad.ubuntu.com email ending up in Launchpad mailbox" [Low,Triaged] https://launchpad.net/bugs/54005
<Ursinha> bigjools, distributionmirror-prober script has failed twice today, is that under soyuz?
<bigjools> Ursinha: registry
<Ursinha> bigjools, thanks
<Ursinha> EdwinGrubbs, do you know what's going on with distributionmirror-prober script, that failed twice today?
<EdwinGrubbs> Ursinha: no, I don't know anything about it. can you point me at the error messages? I'll look into it.
<Ursinha> EdwinGrubbs, sure, I'll do that as soon as the meeting finishes, thanks!
<Ursinha> we have no critical bugs! yay!
<Ursinha> I'm moving on
<Ursinha> thanks guys
<Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
* Ursinha hands Chex the mic
<Ursinha> Chex, stage is empty, c'mon
<Chex> hello everyone
<danilos> hello Chex
<Chex> it was a quiet week LP-wise as us LOSA's were in release prep all week.
<Chex> I just have these incidents to report:
<Chex>  - LP incidents of note: : 22-Apr: CP 9194 to librarian : 25-Apr: each app server on wampee using > 1.1G RSS - restarted
<Chex> Does Anyone have any questions or comments for us?
<Ursinha> anyone?
<Ursinha> thanks Chex 
<Ursinha> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<Ursinha> I sent one email to stub asking for the report
<Ursinha> next :)
<Ursinha> [TOPIC] * QA stats of the week
<MootBot> New Topic:  * QA stats of the week
<Ursinha> launchpadlib: 10
<Ursinha> launchpad-buildd: 10
<Ursinha> lp-dev-utils: 9
<Ursinha> launchpad-dev-moin-theme: 6
<Ursinha> launchpad-code: 5
<Ursinha> soyuz: 4
<Ursinha> malone: 4
<Ursinha> launchpad-registry: 3
<Ursinha> lp-qa-tools: 3
<Ursinha> lpbuildbot: 2
<Ursinha> trac-launchpad-migrator: 2
<Ursinha> launchpad-news-wordpress-theme: 2
<Ursinha> launchpad-help-moin-theme: 2
<Ursinha> launchpad-cscvs: 2
<Ursinha> launchpad-loggerhead: 1
<Ursinha> launchpad-documentation: 1
<Ursinha> tickcount: 1
<Ursinha> so I guess we're doing pretty well triaging, those numbers look better every week
<Ursinha> I'll triage the lp-qa-tools ones
<bigjools> what are those numbers?
<Ursinha> bigjools, untriaged bugs for each of those projects
<Ursinha> bigjools, soyuz has 4 :)
<Ursinha> [action] Ursinha to triage lp-qa-tools bugs
<MootBot> ACTION received:  Ursinha to triage lp-qa-tools bugs
<bigjools> QA stats evokes something else for me
<Ursinha> bigjools, I agree with you :) why took you so long to say that? :)
<bigjools> I was trying to say it in a way that wouldn't offend :)
<bigjools> not sure I didn't fail
<Ursinha> Untested items per team:
<Ursinha> Code: 4
<Ursinha> Foundations: 4
<Ursinha> Registry: 2
<Ursinha> Bugs: 6
<Ursinha> Translations: 3
<Ursinha> Soyuz: 7
<Ursinha> Strategy: 1
<bigjools>  /o\
<Ursinha> bigjools, no, that's ok :) I'm asking because I guess we're having this section of the meeting for over a month now
<Ursinha> bigjools, not sure that helped you :P
<bigjools> :)
<Ursinha> I'll try to come up with a better way to use that info here
<Ursinha> but I can say that all teams are pretty good comparing to past cycles :)
<Ursinha> I'm moving on then
<Ursinha> bigjools, thanks :)
<Ursinha> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<Ursinha> None
<Ursinha> anyone wants to say something before this meeting closes?
<Ursinha> ok then :)
<Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
<Ursinha> #endmeeting
<MootBot> Meeting finished at 11:35.
<Ursinha> thanks all, and happy Ubuntu release everyone!