1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <sinzui> me
   8 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   9 <henninge> me
  10 <rockstar> ni!
  11 <bigjools> me
  12 <henninge> rockstar: that's plural, 'mi' is singular.
  13 <Ursinha> me
  14 <intellectronica> me
  15 <matsubara> Chex, hi
  16 <matsubara> gary_poster, hi
  17 <rockstar> henninge, :)  Making a Monty Python reference.
  18 <gary_poster> me
  19 <gary_poster> thanks
  20 <henninge> oh, don't know that one.
  21 <matsubara> apologies from stub
  22 <henninge> rockstar: Mine was an Esperanto reference
  23 <matsubara> [TOPIC] Agenda 
  24 <MootBot> New Topic:  Agenda
  25 <matsubara>  * Actions from last meeting
  26 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  27 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  28 <matsubara>  * DBA report (stub)
  29 <matsubara>  * Proposed items
  30 <matsubara> [TOPIC] * Actions from last meeting
  31 <MootBot> New Topic:  * Actions from last meeting
  32 <matsubara>     * barry to continue debug on bug 403606
  33 <matsubara>     * intellectronica to take a look or find someone to take a look on bug 408738
  34 <matsubara>     * matsubara to chase ubottu maintainer to fix regexp used to identify oopses(i.e. oops-tools shouldn't trigger the bot)
  35 <matsubara>         * filed bug 420032 and talked to jussi01 on #ubuntu-bots about it.
  36 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  37 <ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
  38 <ubottu> Launchpad bug 420032 in ubuntu-bots "don't trigger OOPS URL when 'oops-tools' is mentioned" [Undecided,New] https://launchpad.net/bugs/420032
  39 <Chex> matsubara: hi
  40 <matsubara> ubottu, oops-tools
  41 <ubottu> Sorry, I don't know anything about oops-tools
  42 <matsubara> oops-tools
  43 <matsubara> cool. looks like that one was fixed :-)
  44 <Ursinha> \o/
  45 <gary_poster> heh
  46 <Ursinha> ubottu, good boy
  47 <ubottu> Sorry, I don't know anything about good boy
  48 <gary_poster> lol
  49 <intellectronica> matsubara: it's on the radar, but we haven't yet had the chance to fix it. as you can imagine, v3.0 UI is keeping everyone very busy
  50 <matsubara> sinzui, any news about 403606?
  51 <matsubara> intellectronica, right. shall I keep it in the action items list?
  52 <sinzui> no
  53 <intellectronica> matsubara: yes, please
  54 <sinzui> I ask my team to only work on UI
  55 <Ursinha> intellectronica, thanks
  56 <matsubara> thanks intellectronica and sinzui 
  57 <matsubara> [action] * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  58 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  59 <MootBot> ACTION received:  * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  60 <sinzui> matsubara: Since there was nothing obviously wrong, we cannot judge the scope of the issue
  61 <matsubara> [action]  * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
  62 <ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
  63 <MootBot> ACTION received:   * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
  64 <matsubara> sinzui, ok, let's keep on the radar though. that's the most frequent oops with have
  65 <matsubara> right Ursinha ?
  66 <Ursinha> right
  67 <matsubara> ok, thanks
  68 <matsubara> let's move on
  69 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  70 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  71 <Ursinha> that's me
  72 <matsubara> Ursinha, stage is yours
  73 <Ursinha> bugs 4-08738 and 4-03606 already mentioned and discussed
  74 <Ursinha> I have bug 422560 for answers, but rockstar told me yesterday he's on it, right rockstar? could you update the bug, pretty please?
  75 <ubottu> Launchpad bug 422560 in launchpad-answers ""Create a FAQ" link fails - Oops! error page" [High,Triaged] https://launchpad.net/bugs/422560
  76 <Ursinha> gary_poster, spm reported this one yesterday: bug 422960, seems to be foundations, what do you say?
  77 <ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960
  78 <rockstar> Ursinha, yup, getting ready to submit for review.
  79 <gary_poster> ...looking...
  80 <Ursinha> rockstar, cool, can you add your precious comments to the bug report? :)
  81 <matsubara> I asked some info from spm on that foundations bug Ursinha 
  82 <Ursinha> matsubara, I see
  83 <rockstar> Ursinha, yes ma'am
  84 <Ursinha> thanks rockstar :)
  85 <Ursinha> ah
  86 <Ursinha> rockstar, bug 401631 is the one about +branches timeout, it's committed on edge but I see lots of timeouts on lpnet daily
  87 <matsubara> Ursinha, how about: 497 ValueError: Invalid boundary in multipart form: '' ?
  88 <Ursinha> maybe worth a cherrypick?
  89 <ubottu> Launchpad bug 401631 in launchpad-code "Repeated timeouts when viewing profile page on edge" [High,Fix committed] https://launchpad.net/bugs/401631
  90 <gary_poster> Ursinha, I'll say yes at least on a first take.  We'll at least do some investigation.
  91 <rockstar> Ursinha, I'll chat with thumper about it.
  92 <gary_poster> and either take it or figure out who else should take it :-)
  93 <gary_poster> so it's ours now, anyway
  94 <Ursinha> thanks gary_poster
  95 <matsubara> Ursinha, I'm pretty sure the timeouts on person +branches was CP already
  96 <Ursinha> matsubara, I still see timeouts
  97 <matsubara> Ursinha, it was CP on the 1s. did the CP at least alleviated the problem?
  98 <matsubara> Ursinha, and how about the 497 ValueErrors on edge?
  99 <Ursinha> matsubara, wait man :)
 100 <Ursinha> it seems the problem is better now
 101 <matsubara> I am
 102 <Ursinha> I'll keep watching the next days
 103 <Ursinha> about the value error
 104 <Ursinha> I was going to ask rockstar if that's a code problem
 105 <Ursinha> rockstar, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1341EA197
 106 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1341EA197
 107 <matsubara> [action] ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
 108 <MootBot> ACTION received:  ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
 109 <rockstar> Ursinha, huh, that's a weird and informationless oops.
 110 <Ursinha> rockstar, agreed about the informationless
 111 <matsubara> [action] ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
 112 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107
 113 <MootBot> ACTION received:  ursinha to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
 114 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107
 115 <Ursinha> my only hint was the path
 116 <Ursinha> I'll file a bug
 117 <matsubara> thanks Ursinha and rockstar 
 118 <Ursinha> ok
 119 <Ursinha> we have one critical bug, already fix committed
 120 <Ursinha> about the failing script, rosetta-poimport, henninge, do you know what's causing that to fail?
 121 <matsubara> Ursinha, isn't that the one that danilo replied to?
 122 <henninge>  Ursinha: yes, it's got to do with sending emails.
 123 <matsubara> s/danilo/jtv and danilo/
 124 <Ursinha> matsubara, yes, but I don't seem to see a conclusion there
 125 <henninge> Ursinha: jtv found the bug and the fix is on the way in
 126 <Ursinha> henninge, awesome :) thanks!
 127 <matsubara> cool
 128 <henninge> Ursinha: we are going to CP it
 129 <Ursinha> henninge, great, will jtv ask for a CP?
 130 <henninge> yes
 131 <henninge> I believe so, at least.
 132 <Ursinha> henninge, that's good
 133 <matsubara> [action] matsubara to chase jtv about CP to fix rosetta-poimport script failure
 134 <MootBot> ACTION received:  matsubara to chase jtv about CP to fix rosetta-poimport script failure
 135 <Ursinha> thakn you :)
 136 <Ursinha> *thank
 137 <Ursinha> thanks all
 138 <Ursinha> matsubara, go ahead
 139 <matsubara> hmm actually I think we should swap some action items Ursinha :-)
 140 <matsubara> but I'll talk about it after the meeting
 141 <matsubara> thanks Ursinha and everyone
 142 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 143 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 144 <matsubara> Chex, ?
 145 <mbarnett> sorry, i think i might have been distracting him with a question.
 146 <Chex> sorry
 147 <Chex> here is our report for the week:
 148 <Ursinha> losas.. :)
 149 <Chex> Increasing number of 5XX errors in apache pretty much entirely from msn
 150 <Chex> bot on translations pages
 151 <Chex> Bug about missing OOPSes - bug 422960
 152 <ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960
 153 <Chex> Codebrowse is still needing restarting pretty regularly per the Incident Log
 154 <Chex> and thats all from us, unless there are any questions?
 155 <matsubara> Chex, we (as in foundations team) will investigate the 500s. I don't think we should be logging OOPS for those as they don't seem to be hitting the app server. It looks like an issue with the load balancer or apache
 156 <rockstar> Chex, I've looked into that.  There is more to look at, but I think we have some hints on wtf is going on.
 157 * mbarnett cheers 
 158 <gary_poster> Chex, I'll follow up with you all on the 5XX later today.
 159 <Chex> great, thanks for the response/feedback
 160 <matsubara> Chex, would be awesome if you could add a request that generated one of the 500 to the bug report
 161 <gary_poster> +1
 162 <matsubara> [action] chex to trawl logs and add request that caused 500 error to bug 422960
 163 <ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,Incomplete] https://launchpad.net/bugs/422960
 164 <MootBot> ACTION received:  chex to trawl logs and add request that caused 500 error to bug 422960
 165 * andrea-bs has quit (Remote closed the connection)
 166 <Chex> matsubara: ok, I will look into that for you
 167 <matsubara> thanks Chex 
 168 <matsubara> let's move on
 169 <Chex> matsubara: welcome
 170 <matsubara> [TOPIC] * DBA report (stub)
 171 <MootBot> New Topic:  * DBA report (stub)
 172 <matsubara> so stuart is not around. I'll email him about the dba report
 173 <matsubara> [action] matsubara to email stub about dba report
 174 <MootBot> ACTION received:  matsubara to email stub about dba report
 175 <matsubara> [TOPIC] * Proposed items
 176 <MootBot> New Topic:  * Proposed items
 177 <matsubara> no proposed items
 178 <matsubara> so I think that's all for today
 179 <matsubara> anything else before I close?
 180 <Ursinha> nothing here
 181 <matsubara> ok, so be it
 182 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 183 <matsubara> #endmeeting 
 184 <MootBot> Meeting finished at 10:27.