1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <gary_poster> me
   8 <rockstar> ni! ni!
   9 <danilos> me
  10 <allenap> me
  11 <sinzui> me
  12 <matsubara> Chex, bigjools, hi
  13 <bigjools> me
  14 <Chex> Chex: hello
  15 <matsubara> ok, everyone is here.
  16 <Chex> .. /o\ err, hi
  17 <matsubara> apologies from Ursinha and stub
  18 <matsubara> [TOPIC] Agenda 
  19 <MootBot> New Topic:  Agenda
  20 <matsubara>  * Actions from last meeting
  21 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  22 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  23 <matsubara>  * DBA report (stub)
  24 <matsubara>  * Proposed items
  25 <matsubara> [TOPIC] * Actions from last meeting
  26 <MootBot> New Topic:  * Actions from last meeting
  27 <matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
  28 <matsubara> * Chex to follow up with thumper about the multiple git import failures on the importd
  29 <matsubara> * matsubara to file a high/critical bug for OOPS-1430F2574
  30 <matsubara> * matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1427EA45
  31 <matsubara> * matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536
  32 <matsubara>     * emailed Tim about it
  33 <matsubara> * matsubara to talk to TL about not having the LP production meeting anymore or change its format
  34 <matsubara> * matsubara to email QA contacts about next LP prod. meeting at 16UTC
  35 <matsubara>     * emailed the list and QA contacts about this :-)
  36 <matsubara> * matsubara to email losas about their weekly report
  37 <matsubara>     * emailed them requesting that the operation report be sent to the list
  38 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
  39 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1427EA45
  40 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
  41 <matsubara> I talked to salgado about OOPS-1430F2574 and it wasn't necessary to file a bug for that one. it was a one off problem and I'm keeping an eye on oops reports if it shows up again
  42 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
  43 <matsubara> danilos sent the email about the qa-tags experiment
  44 <danilos> matsubara, Ursula is on vacation, I'd like her to give us her PoV as well
  45 <matsubara> Chex, did you sort out the failures in the git import with thumper?
  46 <danilos> matsubara, though, that will likely happen as part of the discussion on the list, so we can probably take the action item off
  47 <matsubara> danilos, right. thanks for starting the discussion
  48 <Chex> matsubara: I followed up with him briefly, but was not able to resolve anything, I need to talk to him again, sorry about that
  49 <danilos> np, it was way overdue
  50 <matsubara> Chex, shall I re-add the action item to the list?
  51 <Chex> matsubara: yes please do
  52 <matsubara> [action] * Chex to follow up with thumper about the multiple git import failures on the importd
  53 <MootBot> ACTION received:  * Chex to follow up with thumper about the multiple git import failures on the importd
  54 <matsubara> ok, thanks Chex 
  55 <matsubara> let's move on
  56 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  57 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  58 <matsubara> bigjools,  https://bugs.edge.launchpad.net/soyuz/+bug/493703
  59 <ubottu> Ubuntu bug 493703 in soyuz "LocationError raised in build page and distribution arch series binary package page" [High,Triaged]
  60 <matsubara> it's targeted to .12, currently not assigned and the cycle will end tomorrow. Any chance of have that one fixed before the holidays? it's generating > 1K OOPS a day (most of them from robots but it's pretty importante nonetheless)
  61 <bigjools> matsubara: zero chance
  62 <matsubara> s/importante/important/ sorry for the portuguese leakage there :-)
  63 <bigjools> heh I am used to it from working with cprov :)
  64 <matsubara> bigjools, :-(
  65 <matsubara> the sad smile is for the zero chance comment btw
  66 <bigjools> yeah, there's another serious problem that is taking precedence.  If by some miracle I get that fixed then we can look at the oopses
  67 <matsubara> hmm that's the top OOPS we have
  68 <bigjools> gar sorry
  69 <bigjools> when will the pain end this week
  70 <matsubara> oh, the retry dep thingie?
  71 <bigjools> yep
  72 <matsubara> right. ok then
  73 <matsubara> gary_poster, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403618
  74 <ubottu> Ubuntu bug 403618 in launchpad-foundations "Launchpad should return a 404 instead of ForbiddenAttribute OOPS" [High,Triaged]
  75 <matsubara> gary_poster, same thing, that one is happening quite frequently. any chance of landing a fix before the holidays?
  76 <gary_poster> matsubara: holidays, yes, next release, no
  77 <gary_poster> where yes is "any chance" :-)
  78 <gary_poster> I suppose it can be an RC then
  79 <gary_poster> I mean CP
  80 <matsubara> gary_poster, all right. as long as they disappear from the OOPS summaries, it's good :-)
  81 <gary_poster> :-) understood
  82 <matsubara> gary_poster, could you take a look at https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 ?
  83 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
  84 <matsubara> it's a timeout error on the api
  85 <matsubara> I'm not sure if it's just regular timeouts, if they do, then I'd need to update oops-tools to handle those just like any other timeouts
  86 <matsubara> currently they show up in the exceptions section
  87 <gary_poster> matsubara: yes, it's another timeout
  88 <matsubara> rockstar, OOPS-1438EA844
  89 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1438EA844
  90 <matsubara> gary_poster, so just a matter of moving that kind of exception to the right section in the summaries? 
  91 <gary_poster> matsubara: i.e., this is something that should be addressed by bugs, notfoundatons/leonardr
  92 * rockstar looks
  93 <matsubara> gary_poster, ok, looks like a time out using the dupe finder
  94 <matsubara> but using the API
  95 <matsubara> so, I'll talk to the bugs team about it and sort it out (and file a bug to have oops-tools updated to handle it appropriately)
  96 <gary_poster> matsubara: right.  the problem probably needs to be addressed in lp.bugs.model.bugtask, line 571, in findSimilarBugs
  97 <matsubara> sinzui, https://bugs.edge.launchpad.net/launchpad-registry/+bug/495051
  98 <ubottu> Ubuntu bug 495051 in launchpad-registry "UnboundLocalError editing proposed team membership" [High,In progress]
  99 <sinzui> 'nough said
 100 <matsubara> allenap, I have a few timeout OOPSes on +filebug. are you interested in those? I know gmb just landed code to make it async so maybe it's just a matter of waiting for that and see how things will behave
 101 <matsubara> sinzui, indeed! thanks dude!
 102 <danilos> matsubara, how common is eg. bug 493703 in OOPSes? it looks reasonably simple to solve that somebody outside soyuz can fix it? bigjools, is my assessment wrong?
 103 <ubottu> Launchpad bug 493703 in soyuz "LocationError raised in build page and distribution arch series binary package page" [High,Triaged] https://launchpad.net/bugs/493703
 104 <allenap> matsubara: gmb's async stuff probably won't make the timeouts any different, it's just that the user won't be so affected.
 105 <bigjools> danilos: noodles is going to look into it
 106 <matsubara> [action] matsubara to talk to bugs team about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 and file a bug on oops-tools to handle LaunchpadTimeoutError correctly
 107 <MootBot> ACTION received:  matsubara to talk to bugs team about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 and file a bug on oops-tools to handle LaunchpadTimeoutError correctly
 108 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
 109 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
 110 <allenap> matsubara: The URL will change slightly, to +filebug-inline-form. Timeouts for this page are far less important.
 111 <danilos> bigjools, ok, if you need help finding someone to work on it (though, looking into it might be most of the work anyway), I'd be happy to give a hand tomorrow (in looking for someone, not doing it :)
 112 <bigjools> danilos: ok thanks :)
 113 <matsubara> danilos, thanks. we have around 1K of those a day (mostly from bots triggering it)
 114 <danilos> matsubara, right, thanks
 115 <matsubara> allenap, all right. yesterday bigkev was trying to file some bugs and couldn't due to timeouts. I wonder if you need more OOPSes to help investigate the issue
 116 <allenap> matsubara: He should be able to file bugs today, because the async dupe-finder is there now. But more OOPS reports are useful, if you have a bug to attach them to?
 117 <matsubara> allenap, cool. I'll add those to the bug report then
 118 <allenap> matsubara: Thanks :)
 119 <matsubara> [action] matsubara to add +filebug timeout oopses to the bug report 
 120 <MootBot> ACTION received:  matsubara to add +filebug timeout oopses to the bug report
 121 <matsubara> rockstar, that SQLObjectNotFound oops is quite strange
 122 <matsubara> rockstar, have you seen it before? looks like it happened only twice 
 123 <rockstar> matsubara, yeah, I'm looking at it.  It probably should have 404'd - Not sure though.
 124 <matsubara> and I was unable to reproduce
 125 <rockstar> I have not seen it before.  It's probably some corrupted bmp somewhere.
 126 <danilos> matsubara, that strikes me as replication-related, but I am no expert :)
 127 <matsubara> rockstar, worth a bug rpeort for that one?
 128 <rockstar> danilos, yeah, that's what I thought.
 129 <rockstar> matsubara, not sure.  I'll look into it, and file one if need be.
 130 <matsubara> danilos, rockstar  yeah, same thought here
 131 <matsubara> rockstar, cool. thanks! I'll let you know if it happens again
 132 <matsubara> we had some script failures since last week
 133 <matsubara> Scripts failed to run: loganberry:allocate-revision-karma, loganberry:flag-expired-memberships
 134 <matsubara> sinzui, ^ I think that one is yours?
 135 <matsubara> the retry depwait script failure is being worked on by bigjools 
 136 <sinzui> matsubara: We have had intermittent timing issues because of long processes
 137 <bigjools> floundering on
 138 <sinzui> matsubara: there are no errors and the script do run when they get their turn
 139 <matsubara> sinzui, ok, so that means that the failures on mizuho:librarian-gc and loganberry:karma-update, loganberry:allocate-revision-karma, loganberry:launchpad-stats, loganberry:expire-questions, loganberry:productreleasefinder, loganberry:update-cache, loganberry:launchpad-targetnamecacheupdate are probably related to that?
 140 <matsubara> I guess so, since the last failures for those were 2 days ago
 141 <matsubara> we have only one critical bug which bigjools is on it. 
 142 <sinzui> matsubara: right. I do not investigate a failure to run for 24 hours after the notice because ANOTHER process is responsible for that. When all scripts fail, I might investigate withing 24 hours
 143 <matsubara> I see. all right then. thanks sinzui 
 144 <matsubara> and thanks everyone. let's move on
 145 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 146 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 147 <sinzui> matsubara: and i check if we changed production code in the last 24 hours
 148 <matsubara> Chex, ?
 149 <Chex> hello everyone, a rport focused on the LP rollout:
 150 <matsubara> wow, nice timing :-)
 151 <Chex> - The LP 3.1.11 rollout was last week, and there is a upcoming 'short' LP rollout next week.
 152 <Chex> - 3.1.11 roll-out took 2 days, due to some problems with the rollout
 153 <Chex>           process. We are working to address these issues for next time.
 154 <Chex>     Steps we are taking to improve the process are:
 155 <Chex>         : moving to build centrally before pushing code out to speed up pushing and building of code
 156 <Chex>         : investigating less error prone ways (and quicker ways) of switching to read-only mode
 157 <Chex>         : ensuring we're not interrupted by other DB jobs on other servers in the cluster that block the DB upgrade
 158 <Chex> and thats all for this week, questions/comments, anyone??
 159 <matsubara> Chex, thanks
 160 * noodles775 has quit (Read error: 110 (Connection timed out))
 161 <Chex> matsubara: your welcome
 162 <matsubara> [TOPIC] * DBA report (stub)
 163 <MootBot> New Topic:  * DBA report (stub)
 164 <danilos> Chex, yes, are we getting any of this for 3.1.12?
 165 <matsubara> oops, sorry, go ahead danilos 
 166 <matsubara> [action] matsubara to email stub about the DBA report
 167 * noodles775 (n=miken@canonical/launchpad/noodles775) has joined #launchpad-meeting
 168 <MootBot> ACTION received:  matsubara to email stub about the DBA report
 169 <Chex> danilos: yes, most of those items I listed should make it into the 3.1.12 release, I believe
 170 <danilos> Chex, ok, cool, that sounds great then, but there's always potential for failure with new features like that; I'll try to keep an eye on that :)
 171 <Chex> danilos: ok, great, we appreciate all and any eyeballs on the process
 172 <danilos> Chex, fwiw, I'll be doing a release manager rotation, it's not that I don't trust our lovely LOSA team :)
 173 <danilos> bigjools, (add a link to the image :)
 174 <bigjools> http://people.canonical.com/~ed/losa-team.png
 175 <MootBot> LINK received:  http://people.canonical.com/~ed/losa-team.png
 176 <matsubara> LOL
 177 <danilos> thank you, we can move on :)
 178 <matsubara> sorry, got very distracted by that picture hehe
 179 <matsubara> [TOPIC] * Proposed items
 180 <MootBot> New Topic:  * Proposed items
 181 <matsubara> there's no new proposed items
 182 <matsubara> the new meeting time seems to work fine for everyone
 183 <matsubara> anything else before I close?
 184 <danilos> And if anyone has any issues that may need tracking, please ping me as the release manager for 3.1.12. Thank you.
 185 <bigjools> danilos, my hero
 186 * salgado-lunch is now known as salgado
 187 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 188 <matsubara> #endmeeting 
 189 <MootBot> Meeting finished at 10:37.