## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<gary_poster> me
<rockstar> ni! ni!
<danilos> me
<allenap> me
<sinzui> me
<matsubara> Chex, bigjools, hi
<bigjools> me
<Chex> Chex: hello
<matsubara> ok, everyone is here.
<Chex> .. /o\ err, hi
<matsubara> apologies from Ursinha and stub
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
<matsubara> * Chex to follow up with thumper about the multiple git import failures on the importd
<matsubara> * matsubara to file a high/critical bug for OOPS-1430F2574
<matsubara> * matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1427EA45
<matsubara> * matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536
<matsubara>     * emailed Tim about it
<matsubara> * matsubara to talk to TL about not having the LP production meeting anymore or change its format
<matsubara> * matsubara to email QA contacts about next LP prod. meeting at 16UTC
<matsubara>     * emailed the list and QA contacts about this :-)
<matsubara> * matsubara to email losas about their weekly report
<matsubara>     * emailed them requesting that the operation report be sent to the list
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1427EA45
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
<matsubara> I talked to salgado about OOPS-1430F2574 and it wasn't necessary to file a bug for that one. it was a one off problem and I'm keeping an eye on oops reports if it shows up again
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
<matsubara> danilos sent the email about the qa-tags experiment
<danilos> matsubara, Ursula is on vacation, I'd like her to give us her PoV as well
<matsubara> Chex, did you sort out the failures in the git import with thumper?
<danilos> matsubara, though, that will likely happen as part of the discussion on the list, so we can probably take the action item off
<matsubara> danilos, right. thanks for starting the discussion
<Chex> matsubara: I followed up with him briefly, but was not able to resolve anything, I need to talk to him again, sorry about that
<danilos> np, it was way overdue
<matsubara> Chex, shall I re-add the action item to the list?
<Chex> matsubara: yes please do
<matsubara> [action] * Chex to follow up with thumper about the multiple git import failures on the importd
<MootBot> ACTION received:  * Chex to follow up with thumper about the multiple git import failures on the importd
<matsubara> ok, thanks Chex 
<matsubara> let's move on
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> bigjools,  https://bugs.edge.launchpad.net/soyuz/+bug/493703
<ubottu> Ubuntu bug 493703 in soyuz "LocationError raised in build page and distribution arch series binary package page" [High,Triaged]
<matsubara> it's targeted to .12, currently not assigned and the cycle will end tomorrow. Any chance of have that one fixed before the holidays? it's generating > 1K OOPS a day (most of them from robots but it's pretty importante nonetheless)
<bigjools> matsubara: zero chance
<matsubara> s/importante/important/ sorry for the portuguese leakage there :-)
<bigjools> heh I am used to it from working with cprov :)
<matsubara> bigjools, :-(
<matsubara> the sad smile is for the zero chance comment btw
<bigjools> yeah, there's another serious problem that is taking precedence.  If by some miracle I get that fixed then we can look at the oopses
<matsubara> hmm that's the top OOPS we have
<bigjools> gar sorry
<bigjools> when will the pain end this week
<matsubara> oh, the retry dep thingie?
<bigjools> yep
<matsubara> right. ok then
<matsubara> gary_poster, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403618
<ubottu> Ubuntu bug 403618 in launchpad-foundations "Launchpad should return a 404 instead of ForbiddenAttribute OOPS" [High,Triaged]
<matsubara> gary_poster, same thing, that one is happening quite frequently. any chance of landing a fix before the holidays?
<gary_poster> matsubara: holidays, yes, next release, no
<gary_poster> where yes is "any chance" :-)
<gary_poster> I suppose it can be an RC then
<gary_poster> I mean CP
<matsubara> gary_poster, all right. as long as they disappear from the OOPS summaries, it's good :-)
<gary_poster> :-) understood
<matsubara> gary_poster, could you take a look at https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 ?
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
<matsubara> it's a timeout error on the api
<matsubara> I'm not sure if it's just regular timeouts, if they do, then I'd need to update oops-tools to handle those just like any other timeouts
<matsubara> currently they show up in the exceptions section
<gary_poster> matsubara: yes, it's another timeout
<matsubara> rockstar, OOPS-1438EA844
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1438EA844
<matsubara> gary_poster, so just a matter of moving that kind of exception to the right section in the summaries? 
<gary_poster> matsubara: i.e., this is something that should be addressed by bugs, notfoundatons/leonardr
* rockstar looks
<matsubara> gary_poster, ok, looks like a time out using the dupe finder
<matsubara> but using the API
<matsubara> so, I'll talk to the bugs team about it and sort it out (and file a bug to have oops-tools updated to handle it appropriately)
<gary_poster> matsubara: right.  the problem probably needs to be addressed in lp.bugs.model.bugtask, line 571, in findSimilarBugs
<matsubara> sinzui, https://bugs.edge.launchpad.net/launchpad-registry/+bug/495051
<ubottu> Ubuntu bug 495051 in launchpad-registry "UnboundLocalError editing proposed team membership" [High,In progress]
<sinzui> 'nough said
<matsubara> allenap, I have a few timeout OOPSes on +filebug. are you interested in those? I know gmb just landed code to make it async so maybe it's just a matter of waiting for that and see how things will behave
<matsubara> sinzui, indeed! thanks dude!
<danilos> matsubara, how common is eg. bug 493703 in OOPSes? it looks reasonably simple to solve that somebody outside soyuz can fix it? bigjools, is my assessment wrong?
<ubottu> Launchpad bug 493703 in soyuz "LocationError raised in build page and distribution arch series binary package page" [High,Triaged] https://launchpad.net/bugs/493703
<allenap> matsubara: gmb's async stuff probably won't make the timeouts any different, it's just that the user won't be so affected.
<bigjools> danilos: noodles is going to look into it
<matsubara> [action] matsubara to talk to bugs team about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 and file a bug on oops-tools to handle LaunchpadTimeoutError correctly
<MootBot> ACTION received:  matsubara to talk to bugs team about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 and file a bug on oops-tools to handle LaunchpadTimeoutError correctly
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
<allenap> matsubara: The URL will change slightly, to +filebug-inline-form. Timeouts for this page are far less important.
<danilos> bigjools, ok, if you need help finding someone to work on it (though, looking into it might be most of the work anyway), I'd be happy to give a hand tomorrow (in looking for someone, not doing it :)
<bigjools> danilos: ok thanks :)
<matsubara> danilos, thanks. we have around 1K of those a day (mostly from bots triggering it)
<danilos> matsubara, right, thanks
<matsubara> allenap, all right. yesterday bigkev was trying to file some bugs and couldn't due to timeouts. I wonder if you need more OOPSes to help investigate the issue
<allenap> matsubara: He should be able to file bugs today, because the async dupe-finder is there now. But more OOPS reports are useful, if you have a bug to attach them to?
<matsubara> allenap, cool. I'll add those to the bug report then
<allenap> matsubara: Thanks :)
<matsubara> [action] matsubara to add +filebug timeout oopses to the bug report 
<MootBot> ACTION received:  matsubara to add +filebug timeout oopses to the bug report
<matsubara> rockstar, that SQLObjectNotFound oops is quite strange
<matsubara> rockstar, have you seen it before? looks like it happened only twice 
<rockstar> matsubara, yeah, I'm looking at it.  It probably should have 404'd - Not sure though.
<matsubara> and I was unable to reproduce
<rockstar> I have not seen it before.  It's probably some corrupted bmp somewhere.
<danilos> matsubara, that strikes me as replication-related, but I am no expert :)
<matsubara> rockstar, worth a bug rpeort for that one?
<rockstar> danilos, yeah, that's what I thought.
<rockstar> matsubara, not sure.  I'll look into it, and file one if need be.
<matsubara> danilos, rockstar  yeah, same thought here
<matsubara> rockstar, cool. thanks! I'll let you know if it happens again
<matsubara> we had some script failures since last week
<matsubara> Scripts failed to run: loganberry:allocate-revision-karma, loganberry:flag-expired-memberships
<matsubara> sinzui, ^ I think that one is yours?
<matsubara> the retry depwait script failure is being worked on by bigjools 
<sinzui> matsubara: We have had intermittent timing issues because of long processes
<bigjools> floundering on
<sinzui> matsubara: there are no errors and the script do run when they get their turn
<matsubara> sinzui, ok, so that means that the failures on mizuho:librarian-gc and loganberry:karma-update, loganberry:allocate-revision-karma, loganberry:launchpad-stats, loganberry:expire-questions, loganberry:productreleasefinder, loganberry:update-cache, loganberry:launchpad-targetnamecacheupdate are probably related to that?
<matsubara> I guess so, since the last failures for those were 2 days ago
<matsubara> we have only one critical bug which bigjools is on it. 
<sinzui> matsubara: right. I do not investigate a failure to run for 24 hours after the notice because ANOTHER process is responsible for that. When all scripts fail, I might investigate withing 24 hours
<matsubara> I see. all right then. thanks sinzui 
<matsubara> and thanks everyone. let's move on
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<sinzui> matsubara: and i check if we changed production code in the last 24 hours
<matsubara> Chex, ?
<Chex> hello everyone, a rport focused on the LP rollout:
<matsubara> wow, nice timing :-)
<Chex> - The LP 3.1.11 rollout was last week, and there is a upcoming 'short' LP rollout next week.
<Chex> - 3.1.11 roll-out took 2 days, due to some problems with the rollout
<Chex>           process. We are working to address these issues for next time.
<Chex>     Steps we are taking to improve the process are:
<Chex>         : moving to build centrally before pushing code out to speed up pushing and building of code
<Chex>         : investigating less error prone ways (and quicker ways) of switching to read-only mode
<Chex>         : ensuring we're not interrupted by other DB jobs on other servers in the cluster that block the DB upgrade
<Chex> and thats all for this week, questions/comments, anyone??
<matsubara> Chex, thanks
* noodles775 has quit (Read error: 110 (Connection timed out))
<Chex> matsubara: your welcome
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<danilos> Chex, yes, are we getting any of this for 3.1.12?
<matsubara> oops, sorry, go ahead danilos 
<matsubara> [action] matsubara to email stub about the DBA report
* noodles775 (n=miken@canonical/launchpad/noodles775) has joined #launchpad-meeting
<MootBot> ACTION received:  matsubara to email stub about the DBA report
<Chex> danilos: yes, most of those items I listed should make it into the 3.1.12 release, I believe
<danilos> Chex, ok, cool, that sounds great then, but there's always potential for failure with new features like that; I'll try to keep an eye on that :)
<Chex> danilos: ok, great, we appreciate all and any eyeballs on the process
<danilos> Chex, fwiw, I'll be doing a release manager rotation, it's not that I don't trust our lovely LOSA team :)
<danilos> bigjools, (add a link to the image :)
<bigjools> http://people.canonical.com/~ed/losa-team.png
<MootBot> LINK received:  http://people.canonical.com/~ed/losa-team.png
<matsubara> LOL
<danilos> thank you, we can move on :)
<matsubara> sorry, got very distracted by that picture hehe
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> there's no new proposed items
<matsubara> the new meeting time seems to work fine for everyone
<matsubara> anything else before I close?
<danilos> And if anyone has any issues that may need tracking, please ping me as the release manager for 3.1.12. Thank you.
<bigjools> danilos, my hero
* salgado-lunch is now known as salgado
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:37.