## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<gary_poster> me
<mrjazzcat> me
<Ursinha> me me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<matsubara> sinzui, Chex, rockstar, bigjools, allenap, danilos: hi
<bigjools> \o
<allenap> me
<rockstar> hi
<sinzui> me
<Chex> hello
<danilos> me
<matsubara> ok, everyone is here.
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * matsubara to discuss with foundations bug 403281 and set importance on it
<matsubara>         * discussed with Gary and it's now triaged and an importance set.
<matsubara>     * rockstar to triage bug 531687
<matsubara>     * matsubara to discuss bug 531965 with foundations as well
<matsubara>         * discussed with Gary and it's now triaged and an importance set.
<matsubara>     * rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts
<matsubara>         * <rockstar> matsubara, abentley is already figuring out what's going on with those scripts.
<matsubara>     * rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it
<matsubara>     * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
* flacoste (~francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
<ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Low,Fix committed] https://launchpad.net/bugs/403281
<ubottu> Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [High,Triaged] https://launchpad.net/bugs/531687
<ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Low,Invalid] https://launchpad.net/bugs/531965
<matsubara> I didn't add the count of untriaged bugs per team to the oops section yet
<matsubara> [action] * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
<MootBot> ACTION received:  * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
<Ursinha> yes sir
<matsubara> rockstar, any news about the OOM error? did you get back to losas about it?
<matsubara> I see that 531687 was properly triaged and an importance set. thanks for that rockstar 
<rockstar> matsubara, others on my team are looking into it.  We think it's probably just another scalability issue.
<matsubara> cool. thanks rockstar 
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> I have 3 bugs for today, 2 of them were already dealt with
<matsubara> bug 535071 still need some foundations love
<ubottu> Launchpad bug 535071 in launchpad-foundations "Better error handling when librarian is offline" [Undecided,New] https://launchpad.net/bugs/535071
<matsubara> gary_poster, can you take a look ^? 
<matsubara> fwiw, the other two are bugs 535078, 535672
<ubottu> Launchpad bug 535078 in launchpad-foundations "InvalidURIError raised accessing an API url" [High,Triaged] https://launchpad.net/bugs/535078
<ubottu> Launchpad bug 535672 in malone "TransactionRollbackError using the api to find users affected with dupes" [Undecided,Won't fix] https://launchpad.net/bugs/535672
<gary_poster> looking (how did I miss this? :-/ )
<matsubara> thanks gary_poster 
<matsubara> we have 4 critical bugs, 3 fix committed and 1 in progress.
<bigjools>  /o\
<matsubara> for the failing scripts
<matsubara> we have two failing frequently: productreleasefinder and oops-pruner
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=pruner
<gary_poster> heh
<Ursinha> ubottu: there, there
<matsubara> I replied to one of the oops pruner failure email but didn't get a reply yet.
<matsubara> Chex, could you take a look in the oops pruner script logs and check why it's failing?
<Chex> matsubara: yes sure, I can look at that
<matsubara> Chex, please reply to one of the failures emails with the traceback or file a bug
<matsubara> [action] Chex to verify why oops-pruner is failing and reply to the failure email with more information.
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=pruner
<MootBot> ACTION received:  Chex to verify why oops-pruner is failing and reply to the failure email with more information.
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=pruner
<matsubara> sinzui, how about the prf script?
<matsubara> been failing on a daily basis for the past week
<sinzui> It is always running 2 hours late
<sinzui> It always runs, just late
<sinzui> I think we need to consider changing the threshold, or changing the order
<matsubara> sinzui, so we should adjust the monitoring script to take that into account, shouldn't we?
<matsubara> [action] sinzui to work with losas to set the increase threshold for the product release finder monitoring script.
<MootBot> ACTION received:  sinzui to work with losas to set the increase threshold for the product release finder monitoring script.
<matsubara> I think that's all for today's oops/scripts/critical section. thanks everyone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Chex> hello everyone, here is this weeks report:
<Chex> - We are in the process of working on migrating to using SSO run by ISD branded for LP (open-ID single signon) in
<Chex>         Launchpad.
<Chex> - The LP DBs on one of the slave DB servers may need to be rebuilt/repacked as we're getting short on
<Chex>         space on that server.
<Chex> - LP incidents of note:
<Chex>         ; LP Cherry-picks: CP to crowberry/codehost; revno 9082
<Chex>         ; There was a buildbot error that stopped merging from db-devel to db-stable for a few days, which has
<Chex>                 been fixed.
<Chex> and thats all for us.  Any questions/comments??
<matsubara> Chex, I take you're working with stub to have the db rebuilding done?
<Chex> oh, and the buildbot merge error seems to be happening again, so this still needs to be investigated.
<Chex> matsubara: yes I believe we will work with stub on the db rebuild.
<matsubara> rockstar, can you take that? that's a build engineer thing, right?
<rockstar> matsubara, I'm not build engineer anymore.
<matsubara> rockstar, oh, who's the build engineer for this cycle?
<rockstar> matsubara, I don't think there is one.
<matsubara> I guess it'll fallback to foundations then. gary_poster can you work with losas re: the buildbot issue above?
<gary_poster> matsubara: yes
<matsubara> cool. thanks.
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> sorry, forgot to thank Chex for his report. thank you Chex!
<matsubara> dba report was sent to the list. any question please follow up there
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no new proposed items
<matsubara> and I guess that's all for today folks. anything else before I close?
<Chex> matsubara: no problem!
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:23.