#format IRC
<matsubara> moo
<sinzui> cluck
<gary_poster> hee-haw
<danilos> meow
<Ursinha> hello
<Ursinha> #startmeeting
<Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<MootBot> Meeting started at 11:01. The chair is Ursinha.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<Ursinha> [TOPIC] Roll Call
<Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<MootBot> New Topic:  Roll Call
<Ursinha> meeee
<sinzui> me
<gary_poster> smee
<matsubara> me
<Ursinha> rockstar, Chex, hello :)
<Ursinha> I see no soyuz people :(
* adeuring (~abel@93.217.153.185) has joined #launchpad-meeting
<Ursinha> I know we'll have no bugs people here, so ok
<Ursinha> well adeuring is here, but not sure if it's for our meeting :)
<Ursinha> I'll move on and expect people to show up
<Ursinha> [TOPIC] Agenda
<Ursinha>  * Actions from last meeting
<Ursinha>  * Oops report & Critical Bugs & Broken scripts
<Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
<Ursinha>  * DBA report (stub)
<Ursinha>  * QA stats of the week
<Ursinha>  * Proposed items
<MootBot> New Topic:  Agenda
<adeuring> Ursinha: yes, that's why I'm here, but I have another (mumble) meeting right now.
<Ursinha> [TOPIC] * Actions from last meeting
<Ursinha>  * DONE: matsubara to chase someone from Bugs about https://bugs.edge.launchpad.net/malone/+bug/583385
<MootBot> New Topic:  * Actions from last meeting
* salgado is now known as salgado-lunch
<Ursinha> cool, all done
<Ursinha> adeuring, thanks :)
<ubot5> Error: Could not parse data returned by Launchpad: The read operation timed out (https://launchpad.net/bugs/583385)
<Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> main offenders are bug 504124 (spam processing bug?) and bug 580181
<ubot5> Launchpad bug 504124 in Launchpad Foundations "process-mail.py "need more than 1 value to unpack" (affected: 2, heat: 22)" [High,Triaged] https://launchpad.net/bugs/504124
<ubot5> Launchpad bug 580181 in Soyuz "DistributionSourcePackageRelease page still oopsing with NotOneError/LocationError (affected: 1, heat: 6)" [High,Triaged] https://launchpad.net/bugs/580181
<Ursinha> I see the latter should be worked by noodles as soon as he got rid of more urgent problems
* rockstar is here...  had computer problems
* Ursinha hugs rockstar 
<Chex> Ursinha: All us LOSAs are unavailable this week as we are sprinting in Montreal.
<Ursinha> Chex, oh, that's bad, LOSAs' report is so useful
* Ursinha thinks
<sinzui> the process-mail.py bug is a spam attack. Do we want to suppress oopses when the header is malformed?
<Ursinha> this meeting is crippled :/
<Ursinha> sinzui, good question
<Ursinha> gary_poster, we're having annoying high oops counts of 504124, maybe we should do something about it now the counting seems to be raising?
<Ursinha> gary_poster, even what sinzui suggested
<matsubara> sinzui, I think we should not log an oops in that case as you suggested in the bug report
<sinzui> Ursinha, gary_poster. The problem is not the app, but the data so we should gracefully recover I think
<Ursinha> that's my opinion as well
<matsubara> with a try:except when the code is processing the To: header
<gary_poster> Ursinha: yeah, I had that one open already.  If it is really bothering us then let's do it.  Agree with sinzui's approach
<Ursinha> cool
<Ursinha> gary_poster, should I update the bug report with that? 
<sinzui> well, that is now a trivial fix I think
<gary_poster> Ursinha: I'll handle it, thank you :-)
<Ursinha> thanks gary_poster :)
<Ursinha> nice
<Ursinha> thanks all :)
<Ursinha> I'd need a soyuz people here now
<Ursinha> I'll go with the code one then
<Ursinha> rockstar, so, not sure if code team knows about it: OOPS-1607EA658 
<ubot5> https://lp-oops.canonical.com/oops.py/?oopsid=1607EA658
<sinzui> I think the soyuz bug has a deeper root cause...
<sinzui> I looked at it last month and could not see what was happening. I think it goes back to changes made in Jan/Feb
<Ursinha> sinzui, might have, this was "fixed" previously and now shows two different oopses
<rockstar> Ursinha, so, I didn't know about it, but this area still hasn't been tested much...  :/
<rockstar> Ursinha, it won't be released.
<Ursinha> rockstar, oh, right
<Ursinha> rockstar, was that QAed on edge? I'm pointing the oops as it might be a consequence of QAing the feature
<rockstar> Ursinha, well, so no, we can't QA on edge much because we're blocked on two RTs...
<Ursinha> rockstar, I see :/ do you know if the blockage is something that might last for long?
<rockstar> Ursinha, well, the RT was filed as high priority more than a week ago, so your guess is as good as mine.
<Ursinha> rockstar, maybe we should poke some people? :)
* Ursinha cries for a losa
<gary_poster> rockstar, Ursinha, is this ...
<rockstar> gary_poster!
<gary_poster> 39402?
<rockstar> Ursinha, gary_poster is the one who's been chasing for us.
<gary_poster> RT 39402?
<Ursinha> ah, I see
<rockstar> gary_poster, yes, that's it.
<gary_poster> Ursinha, rockstar, mthaddon has said that he will look into it today
<rockstar> gary_poster, ta
<gary_poster> or to not misrepresent: "member:gary: I'll try and take a look at the latter one (the first one is marked resolved) sometime today" :-)
<Ursinha> cool!
<Ursinha> thanks gary_poster and rockstar :)
<gary_poster> np
<Ursinha> rockstar, I'll keep my eyes on that oopses and let you know if something shows up
<Ursinha> thanks
<danilos> rockstar, my suggestion is to also talk to bigjools and lamont to make sure your request is similar to what is being done for "unsupported" packages already
<danilos> rockstar, about that RT at least
<Ursinha> [action] talk to bigjools about the 300 timeouts in the DistroSeries:+index pages everyday, is that expected/known? e.g.: OOPS-1607C1282
<MootBot> ACTION received:  talk to bigjools about the 300 timeouts in the DistroSeries:+index pages everyday, is that expected/known? e.g.: OOPS-1607C1282
<ubot5> https://lp-oops.canonical.com/oops.py/?oopsid=1607C1282
<ubot5> https://lp-oops.canonical.com/oops.py/?oopsid=1607C1282
<matsubara> Ursinha, isn't that the one bigjools replied to recently?
<matsubara> the one that is making an xmlrpc request to another server but is being blocked by a firewall
<Ursinha> matsubara, not sure if that's the same, I guess that one was about builders
<sinzui> Ursinha, That is my oops
<matsubara> ok, just wondering
<Ursinha> matsubara, thanks for that :)
<Ursinha> sinzui, the DistroSeries:+index?
<danilos> Ursinha, yeah, that sounds like registry
<Ursinha> if so, even better, you're here :)
<sinzui> the distroseries-portlet-packaging  is the root cause. I have a branch that add3 3 hours caching to it
<danilos> and fwiw, storm taking too long to serialize stuff
<Ursinha> [action] forget last action
<MootBot> ACTION received:  forget last action
<sinzui> Edwin's definitve SQL fix will not land until next release
<gary_poster> Ursinha, so you know, thumper alerted me last night that this soft timeout is causing timeouts (hard, I assume) on the branch merge proposal page: https://lp-oops.canonical.com/oops.py/?oopsid=1607EA56 .  Therefore I need to make a bug and look into it.
<gary_poster> (Which sucks because pretty much none of know librarian code, but oh well :-) )
<Ursinha> gary_poster, thanks for that
<Ursinha> sinzui, do you have the bug number?
<sinzui> My oops is caused by a monster query. bug 535430 is the root cause
<ubot5> Launchpad bug 535430 in Launchpad Registry "+needs-packaging link times out regularly (affected: 1, heat: 6)" [High,In progress] https://launchpad.net/bugs/535430
<Ursinha> sinzui, right
<sinzui> ^ this is really a model problem that manifests itself on three pages
<Ursinha> sinzui, which ones?
<sinzui> distroseries +index, +needs-packaging +packaging-links  (distroseries-portlet-packaging) plus two full pages
<Ursinha> thank you 
* Ursinha takes notes
<Ursinha> thanks everyone
<sinzui> ^ +packaging, not +packaging-links
* Ursinha fixes her notes
<sinzui> https://edge.launchpad.net/ubuntu/maverick shows that the issue centers on the Upstream packaging data
<Ursinha> sinzui, so your fix will "fix" only part of the problem
<sinzui> yes. timeouts will be reduced by 50% to 80%
<sinzui> That is still disappointing
<Ursinha> that's something
<Ursinha> I guess the model problem won't be fixed (if fixed) anytime soon..
<sinzui> The fix will be too complicated for a CP
<Ursinha> sinzui, do you think that will be fixed soon?
<sinzui> It will be fixed in week 1 on staging
<Ursinha> oh, cool
<Ursinha> better than I imagined
<Ursinha> thanks sinzui 
<Ursinha> I might move on, otherwise the meeting will last forever :)
<Ursinha> we have no critical bugs! \o/
<gary_poster> \o/
<Ursinha> rockstar, upgrade_branches is still failing
<Ursinha> gary_poster, :)
<rockstar> Ursinha, yeah, I'll look into it.  I have some other P1 bugs that need to be fixed first.
<Ursinha> rockstar, sure. Thanks for that. Do you know if there are any bugs opened for that? I can do that!
<rockstar> Ursinha, can you please open one?
<Ursinha> rockstar, sure :)
<Ursinha> rockstar, I'll do that and let you know. Thanks
<rockstar> Ursinha, thanks.
<Ursinha> [action] Ursinha to open a bug for the failing upgrade_branches script
<MootBot> ACTION received:  Ursinha to open a bug for the failing upgrade_branches script
<Ursinha> cool, moving on
<Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Ursinha> losas are sprinting, so no donut for us
<Ursinha> :/
<Ursinha> Chex, may I ask you to send any important news to the list, when you have time? please? :)
* Ursinha values that report
<Ursinha> it's more a wish than anything else, as you're sprinting
<Ursinha> :)
<Ursinha> moving on!
<Ursinha> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<Ursinha> Email asking for the report sent!
<Ursinha> [TOPIC] * QA stats of the week
<MootBot> New Topic:  * QA stats of the week
<Ursinha> untested stuff goes this way:
<Ursinha> Code: 7
<Ursinha> Foundations: 3
<Ursinha> Registry: 4
<Ursinha> Bugs: 3
<Ursinha> Translations: 7
<Ursinha> Soyuz: 3
<Ursinha> Strategy: 0
<Ursinha> I see we're kind of stable re. the number of untested items every week, which is good :)
<Ursinha> right
<Ursinha> about the untriaged bugs
<Ursinha> I've sent one email to lp list asking your opinions about all projects we should take care of
<Ursinha> I don't think pasting the numbers here will be useful now, I'd like you to reply that thread so we know what to do about those
<Ursinha> pretty please?
<danilos> stats for untested stuff seem incorrect for translations :)
<Ursinha> danilos, it was like that 30 mins ago
<danilos> Ursinha, I doubt it, maybe something like 1h or more ago :)
<gary_poster> heh
<Ursinha> danilos, how come? give me a paper bag, please
* gary_poster is huungry :-D
<Ursinha> ok ok
<Ursinha> danilos, you're right, I see only three now
<danilos> Ursinha, oh, the graph is probably slow to update, don't worry about it
<Ursinha> danilos, thanks :)
<Ursinha> so, please, answer the thread!
<Ursinha> :_)
<Ursinha> [TOPIC] * Proposed items
<Ursinha> None from me.
<MootBot> New Topic:  * Proposed items
<Ursinha> anyone wants to say something?
<Ursinha> okay :)
<Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
<Ursinha> #endmeeting
<gary_poster> I'm preparing an email to Ursinha and matsubara about orphan branches