## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<sinzui> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<henninge> me
<rockstar> ni!
<bigjools> me
<henninge> rockstar: that's plural, 'mi' is singular.
<Ursinha> me
<intellectronica> me
<matsubara> Chex, hi
<matsubara> gary_poster, hi
<rockstar> henninge, :)  Making a Monty Python reference.
<gary_poster> me
<gary_poster> thanks
<henninge> oh, don't know that one.
<matsubara> apologies from stub
<henninge> rockstar: Mine was an Esperanto reference
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>     * barry to continue debug on bug 403606
<matsubara>     * intellectronica to take a look or find someone to take a look on bug 408738
<matsubara>     * matsubara to chase ubottu maintainer to fix regexp used to identify oopses(i.e. oops-tools shouldn't trigger the bot)
<matsubara>         * filed bug 420032 and talked to jussi01 on #ubuntu-bots about it.
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
<ubottu> Launchpad bug 420032 in ubuntu-bots "don't trigger OOPS URL when 'oops-tools' is mentioned" [Undecided,New] https://launchpad.net/bugs/420032
<Chex> matsubara: hi
<matsubara> ubottu, oops-tools
<ubottu> Sorry, I don't know anything about oops-tools
<matsubara> oops-tools
<matsubara> cool. looks like that one was fixed :-)
<Ursinha> \o/
<gary_poster> heh
<Ursinha> ubottu, good boy
<ubottu> Sorry, I don't know anything about good boy
<gary_poster> lol
<intellectronica> matsubara: it's on the radar, but we haven't yet had the chance to fix it. as you can imagine, v3.0 UI is keeping everyone very busy
<matsubara> sinzui, any news about 403606?
<matsubara> intellectronica, right. shall I keep it in the action items list?
<sinzui> no
<intellectronica> matsubara: yes, please
<sinzui> I ask my team to only work on UI
<Ursinha> intellectronica, thanks
<matsubara> thanks intellectronica and sinzui 
<matsubara> [action] * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<MootBot> ACTION received:  * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<sinzui> matsubara: Since there was nothing obviously wrong, we cannot judge the scope of the issue
<matsubara> [action]  * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
<ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
<MootBot> ACTION received:   * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
<matsubara> sinzui, ok, let's keep on the radar though. that's the most frequent oops with have
<matsubara> right Ursinha ?
<Ursinha> right
<matsubara> ok, thanks
<matsubara> let's move on
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> that's me
<matsubara> Ursinha, stage is yours
<Ursinha> bugs 4-08738 and 4-03606 already mentioned and discussed
<Ursinha> I have bug 422560 for answers, but rockstar told me yesterday he's on it, right rockstar? could you update the bug, pretty please?
<ubottu> Launchpad bug 422560 in launchpad-answers ""Create a FAQ" link fails - Oops! error page" [High,Triaged] https://launchpad.net/bugs/422560
<Ursinha> gary_poster, spm reported this one yesterday: bug 422960, seems to be foundations, what do you say?
<ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960
<rockstar> Ursinha, yup, getting ready to submit for review.
<gary_poster> ...looking...
<Ursinha> rockstar, cool, can you add your precious comments to the bug report? :)
<matsubara> I asked some info from spm on that foundations bug Ursinha 
<Ursinha> matsubara, I see
<rockstar> Ursinha, yes ma'am
<Ursinha> thanks rockstar :)
<Ursinha> ah
<Ursinha> rockstar, bug 401631 is the one about +branches timeout, it's committed on edge but I see lots of timeouts on lpnet daily
<matsubara> Ursinha, how about: 497 ValueError: Invalid boundary in multipart form: '' ?
<Ursinha> maybe worth a cherrypick?
<ubottu> Launchpad bug 401631 in launchpad-code "Repeated timeouts when viewing profile page on edge" [High,Fix committed] https://launchpad.net/bugs/401631
<gary_poster> Ursinha, I'll say yes at least on a first take.  We'll at least do some investigation.
<rockstar> Ursinha, I'll chat with thumper about it.
<gary_poster> and either take it or figure out who else should take it :-)
<gary_poster> so it's ours now, anyway
<Ursinha> thanks gary_poster
<matsubara> Ursinha, I'm pretty sure the timeouts on person +branches was CP already
<Ursinha> matsubara, I still see timeouts
<matsubara> Ursinha, it was CP on the 1s. did the CP at least alleviated the problem?
<matsubara> Ursinha, and how about the 497 ValueErrors on edge?
<Ursinha> matsubara, wait man :)
<Ursinha> it seems the problem is better now
<matsubara> I am
<Ursinha> I'll keep watching the next days
<Ursinha> about the value error
<Ursinha> I was going to ask rockstar if that's a code problem
<Ursinha> rockstar, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1341EA197
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1341EA197
<matsubara> [action] ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
<MootBot> ACTION received:  ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
<rockstar> Ursinha, huh, that's a weird and informationless oops.
<Ursinha> rockstar, agreed about the informationless
<matsubara> [action] ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107
<MootBot> ACTION received:  ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107
<Ursinha> my only hint was the path
<Ursinha> I'll file a bug
<matsubara> thanks Ursinha and rockstar 
<Ursinha> ok
<Ursinha> we have one critical bug, already fix committed
<Ursinha> about the failing script, rosetta-poimport, henninge, do you know what's causing that to fail?
<matsubara> Ursinha, isn't that the one that danilo replied to?
<henninge>  Ursinha: yes, it's got to do with sending emails.
<matsubara> s/danilo/jtv and danilo/
<Ursinha> matsubara, yes, but I don't seem to see a conclusion there
<henninge> Ursinha: jtv found the bug and the fix is on the way in
<Ursinha> henninge, awesome :) thanks!
<matsubara> cool
<henninge> Ursinha: we are going to CP it
<Ursinha> henninge, great, will jtv ask for a CP?
<henninge> yes
<henninge> I believe so, at least.
<Ursinha> henninge, that's good
<matsubara> [action] matsubara to chase jtv about CP to fix rosetta-poimport script failure
<MootBot> ACTION received:  matsubara to chase jtv about CP to fix rosetta-poimport script failure
<Ursinha> thakn you :)
<Ursinha> *thank
<Ursinha> thanks all
<Ursinha> matsubara, go ahead
<matsubara> hmm actually I think we should swap some action items Ursinha :-)
<matsubara> but I'll talk about it after the meeting
<matsubara> thanks Ursinha and everyone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara> Chex, ?
<mbarnett> sorry, i think i might have been distracting him with a question.
<Chex> sorry
<Chex> here is our report for the week:
<Ursinha> losas.. :)
<Chex> Increasing number of 5XX errors in apache pretty much entirely from msn
<Chex> bot on translations pages
<Chex> Bug about missing OOPSes - bug 422960
<ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960
<Chex> Codebrowse is still needing restarting pretty regularly per the Incident Log
<Chex> and thats all from us, unless there are any questions?
<matsubara> Chex, we (as in foundations team) will investigate the 500s. I don't think we should be logging OOPS for those as they don't seem to be hitting the app server. It looks like an issue with the load balancer or apache
<rockstar> Chex, I've looked into that.  There is more to look at, but I think we have some hints on wtf is going on.
* mbarnett cheers 
<gary_poster> Chex, I'll follow up with you all on the 5XX later today.
<Chex> great, thanks for the response/feedback
<matsubara> Chex, would be awesome if you could add a request that generated one of the 500 to the bug report
<gary_poster> +1
<matsubara> [action] chex to trawl logs and add request that caused 500 error to bug 422960
<ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960
<MootBot> ACTION received:  chex to trawl logs and add request that caused 500 error to bug 422960
* andrea-bs has quit (Remote closed the connection)
<Chex> matsubara: ok, I will look into that for you
<matsubara> thanks Chex 
<matsubara> let's move on
<Chex> matsubara: welcome
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> so stuart is not around. I'll email him about the dba report
<matsubara> [action] matsubara to email stub about dba report
<MootBot> ACTION received:  matsubara to email stub about dba report
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no proposed items
<matsubara> so I think that's all for today
<matsubara> anything else before I close?
<Ursinha> nothing here
<matsubara> ok, so be it
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:27.