1 matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <allenap> me
   8 <mrjazzcat> me
   9 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
  10 <matsubara> Chex, rockstar, bigjools, sinzui, danilos: hi
  11 <matsubara> Ursinha, hi
  12 <rockstar> me
  13 <bigjools> hello
  14 <Ursinha> me
  15 <Ursinha> hi
  16 <danilos> hi
  17 <matsubara> apologies from Gary, he's in a meeting and I'll be the foundations contact for today
  18 <Chex> me
  19 <matsubara> ok, everyone is here
  20 <matsubara> [TOPIC] Agenda 
  21 <MootBot> New Topic:  Agenda
  22 <matsubara>  * Actions from last meeting
  23 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  24 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  25 <matsubara>  * DBA report (stub)
  26 <matsubara>  * Proposed items
  27 <matsubara> [TOPIC] * Actions from last meeting
  28 <MootBot> New Topic:  * Actions from last meeting
  29 <matsubara>     * rockstar to investigate failure on update_branches script and reply to the email.
  30 <matsubara> no new failures since last week, so I guess it sorted out by itself
  31 * rockstar nods
  32 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  33 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  34 <sinzui> me
  35 <Ursinha> it's me
  36 * sinzui did not see the rollcall
  37 <Ursinha> so
  38 <Ursinha> three issues to point
  39 <Ursinha> matsubara: bug 403281 seems an old bug, and I saw a bunch of oopses yesterday caused by this problem, could you set the importance, please?
  40 <ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Undecided,Triaged] https://launchpad.net/bugs/403281
  41 <Ursinha> matsubara: also, I've filed bug 531695, but not sure if that's foundations
  42 <ubottu> Launchpad bug 531695 in libgraph-perl "libgraph-perl v0.81 is not compatible with libheap-perl v0.80" [Undecided,New] https://launchpad.net/bugs/531695
  43 <Ursinha> it seems a rollout issue
  44 <Ursinha> hmm, that's not right
  45 <matsubara> hmm
  46 <matsubara> hehe
  47 <Ursinha> bug 531965
  48 <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Undecided,New] https://launchpad.net/bugs/531965
  49 <Ursinha> oops :)
  50 <matsubara> [action] matsubara to discuss with foundations bug 403281 and set importance on it
  51 <Ursinha> matsubara: what do you say?
  52 <MootBot> ACTION received:  matsubara to discuss with foundations bug 403281 and set importance on it
  53 <ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Undecided,Triaged] https://launchpad.net/bugs/403281
  54 <Ursinha> cool
  55 <sinzui> I think bug 531965 is a soyuz issue
  56 <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Undecided,New] https://launchpad.net/bugs/531965
  57 <Ursinha> sinzui: but that isn't oopsing anymore
  58 <bigjools> no it's not a soyuz issue
  59 <matsubara> there's also bug 403281 which is either Code or foundations
  60 <matsubara> errr
  61 <matsubara> wrong bug number
  62 <Ursinha> sinzui: all occurrences I could find happened during the rollout
  63 <matsubara> I meant bug 531687 
  64 <ubottu> Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [Undecided,New] https://launchpad.net/bugs/531687
  65 <matsubara> rockstar, can you take a look and triage accordingly ^?
  66 <rockstar> matsubara, sure
  67 <rockstar> matsubara, is this part of the meeting really necessary?
  68 <Ursinha> allenap: I see occurrences of https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1524C2016, but not sure this is a known issue
  69 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1524C2016
  70 <matsubara> [action] rockstar to triage bug 531687 
  71 <MootBot> ACTION received:  rockstar to triage bug 531687
  72 <ubottu> Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [Undecided,New] https://launchpad.net/bugs/531687
  73 <allenap> Ursinha: Looking.
  74 <rockstar> We should (and do) keep up with triaging bugs.  I'm not sure that we need to bring it up in a meeting.
  75 <matsubara> rockstar, I think this section of the meeting is the main reason for this meeting :-)
  76 <Ursinha> rockstar: specially after rollouts
  77 <matsubara> may I ask why do you think it's not important?
  78 <Ursinha> :)
  79 <rockstar> matsubara, because you asked me to do something that someone from our team is going to do at some point today anyway.
  80 <rockstar> matsubara, I think the ops report is the most helpful part of this meeting.
  81 <matsubara> yeah, that part is very useful as well, I agree
  82 <matsubara> we bring up the OOPS that we think are important so it won't fall through the cracks
  83 <rockstar> matsubara, do we need to gather everyone together to do that?
  84 <Ursinha> rockstar: but I recall you suggesting us to file bugs for oopses before coming to the meeting, so why exactly you think the oops section is useful?
  85 <rockstar> Ursinha, ops wasn't a typo.  :)  Operations report
  86 <allenap> Ursinha: That reminds me of a known issue. I don't have a bug number for it yet. If it is the same issue then it's proven very difficult to reproduce.
  87 <matsubara> rockstar, well, we already gather everyone together for this meeting. discussing oops issues is just one side of it, along with scripts failing and critical issues
  88 <Ursinha> allenap: that reminds me of a known issue as well, but I couldn't find the bug
  89 <matsubara> the meeting was evolving to the point we're today. from our experience a lot of things end up falling through the cracks if the QA team don't bring them up
  90 <matsubara> but I'm all ears for suggestion to improve the format of the meeting that would be more useful to you
  91 <matsubara> and by you, I mean all of you
  92 <matsubara> :-)
  93 <matsubara> not only rockstar
  94 <rockstar> matsubara, but I ask if we need a meeting to bring them up.  Can't you just bring them up as they happen?
  95 <danilos> rockstar, I think OOPS section is necessary until we get to a point where every team is on top of their OOPSes *without* matsubara and Ursinha
  96 <matsubara> +1 danilos 
  97 <Ursinha> rockstar: I think it's nice to have a main contact to express our concerns :)
  98 <rockstar> danilos, I agree that matsubara and Ursinha are a HUGE lifesaver.  I just wonder if it needs to be part of the meeting.
  99 <Ursinha> I agree with danilos
 100 <rockstar> Ursinha, oh yeah, I'm happy to be the main contact for QA questions.
 101 <danilos> rockstar, let's discuss that at the end of the meeting; Ursinha, matsubara, anyone else interested, let's move that discussion after the regular agenda
 102 <bigjools> I quite value the input from matsubara and Ursinha for the oopses
 103 * rockstar sighs
 104 <matsubara> as danilos pointed out, once the teams are taking care of OOPSes without our help, then I think this section will slowly fade away
 105 <rockstar> bigjools, I don't disagree with you.
 106 <sinzui> I think we have a lot of unowned or shared apps that do not have team
 107 <bigjools> rockstar: I wasn't referring to anything you'd said actually
 108 <rockstar> bigjools, okay.  :)
 109 <danilos> let's go on and come back to this topic later, pleaaseee :)
 110 <sinzui> I am tempted to fix a bug in malone, launchapd-answers, and blueprints because I am tired of reading spam attacks to find my bugs, but who would know about these attacks if we were not collectively seeing all oopses
 111 <matsubara> all right, so Ursinha, anything else? (did I get all the action items right, specially the ones for foundations?)
 112 <Ursinha> matsubara: nothing else in the oops section
 113 <matsubara> [action] matsubara to discuss bug 531965 with foundations as well
 114 <MootBot> ACTION received:  matsubara to discuss bug 531965 with foundations as well
 115 <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Undecided,New] https://launchpad.net/bugs/531965
 116 <matsubara> scripts failing: mpcreationjobs and sendbranchmail just failed
 117 <Ursinha> there are two critical bugs, in progress
 118 <matsubara> rockstar, can you take care of those and reply to the list? is this something that could be caused by the rollout?
 119 <allenap> Ursinha: I can't find out any more about that OOPS now, so I'll investigate later.
 120 <Ursinha> thanks allenap :)
 121 <rockstar> matsubara, sure
 122 <matsubara> [action] rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts
 123 <MootBot> ACTION received:  rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts
 124 <Ursinha> bigjools: will you need a rc to fix bug 530566?
 125 <ubottu> Launchpad bug 530566 in soyuz "cron.germinate still believes lpia is active" [Critical,In progress] https://launchpad.net/bugs/530566
 126 <bigjools> Ursinha: I have one but thanks for reminding me to land that branch :)
 127 <Ursinha> bigjools: :)
 128 <Ursinha> matsubara: there's bug 530354 too that seems approved as well
 129 <ubottu> Launchpad bug 530354 in launchpad-foundations "wadl generation is broken after multiversion code has landed" [Critical,In progress] https://launchpad.net/bugs/530354
 130 <rockstar> matsubara, abentley is already figuring out what's going on with those scripts.
 131 <matsubara> thank you rockstar 
 132 <Ursinha> matsubara: it says "Merged", so that is Fix Committed or even Fix Released? could you check, please?
 133 <matsubara> Ursinha, if it says merged, it's fix committed. I think it's this one: # r9069 [release-critical=thumper][r=gary][ui=none] Cache a static WADL file for every version of the web service. 
 134 <matsubara> probably my script that closes it didn't get to it yet :-)
 135 <Ursinha> matsubara: hehe
 136 <Ursinha> okay
 137 <matsubara> let's move on then
 138 <Ursinha> thanks everyone
 139 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 140 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 141 <matsubara> Chex, the stage is yours
 142 <mbarnett> take it and run!
 143 <Chex> hi everyone
 144 <Chex> Here is the LOSA report for this week:
 145 <Chex> - LP rollout 10.02 took 70 mins longer than scheduled. Some related bugs were:
 146 <Chex>         ; Need more visibility into progress of upgrade.py/fti.py/security.py bug 531833
 147 <ubottu> Launchpad bug 531833 in launchpad-foundations "Need more visibility into the progress of upgrade.py/fti.py/security.py" [High,Triaged] https://launchpad.net/bugs/531833
 148 <Chex>         ; Read-only seemed to leave lots of "idle" and "select waiting" connections to the master and
 149 <Chex>                 slave DBs which possibly blocked the DB upgrade bug 531834
 150 <ubottu> Launchpad bug 531834 in launchpad-foundations "When switching to read-only mode, we're left with lots of "idle/select waiting" connections to the DBs which may be blocking the schema upgrade process" [High,Triaged] https://launchpad.net/bugs/531834
 151 <Chex> - LP incidents of note:
 152 <Chex>         ; Crowberry Out-Of-Memory error: https://pastebin.canonical.com/28737/ 2010-03-04 12:30 UTC
 153 <Chex>           Does anyone have any insight into this issue??
 154 <Chex> and thats the report. anyone have questions or comments?
 155 <matsubara> crowberry is code hosting, right? rockstar ^
 156 <rockstar> matsubara, I'm not sure.  I never remember the names of our boxes...
 157 <matsubara> the pastebin indicates it's a code hosting machine
 158 <matsubara> [action] rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it
 159 <MootBot> ACTION received:  rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it
 160 <Chex> sorry, crowberry is codehosting, yes.
 161 * mrevell has quit (Ping timeout: 252 seconds)
 162 <matsubara> ok, I think we can move on then.
 163 <matsubara> thanks Chex 
 164 <Chex> matsubara: sure
 165 <matsubara> [TOPIC] * DBA report (stub)
 166 <MootBot> New Topic:  * DBA report (stub)
 167 <matsubara> stub will send the report to the list. I'll remind him.
 168 <matsubara> [action] matsubara to remind stub about dba report
 169 <allenap> matsubara: I think he sent it already.
 170 <MootBot> ACTION received:  matsubara to remind stub about dba report
 171 <matsubara> oops
 172 <matsubara> yes, indeed he did. thanks allenap 
 173 <matsubara> [action] cancel last action item about dba report :-)
 174 <MootBot> ACTION received:  cancel last action item about dba report :-)
 175 <matsubara> [TOPIC] * Proposed items
 176 <MootBot> New Topic:  * Proposed items
 177 <matsubara> rockstar, so would you like to discuss a better format for the oops section?
 178 <matsubara> seems to me we have a agreement that the oops section is generally useful and once the teams start to take care of their oopses that section will be unecessary
 179 <danilos> matsubara, we don't have agreement that it will be unnecessary :)
 180 <danilos> matsubara, sinzui has raised an interesting point
 181 <danilos> matsubara, there's a whole bunch of problems which don't belong in any particular team domain
 182 <danilos> matsubara, and then there are also those which appear only for one or two teams, but might be global
 183 <matsubara> right, so we'll continue to bring up those during this meeting
 184 <danilos> I am hoping we'll get to a point where OOPS section will be about those
 185 <matsubara> yep. agreed. 
 186 <rockstar> I think we're pretty good at saying "this doesn't look like a <app> issue.  Maybe ask <other-app-dev>"
 187 <danilos> and prerequisite for not reporting any OOPSes in a meeting is that both teams handle them themselves and that teams are good at bug triage
 188 <rockstar> If there's a bug filed on an oops, it'll get triaged.  That should be a team's responsibility, and if the team isn't getting to it, then we need to evaluate that.
 189 <rockstar> Code has been going out of our way to make sure we get bugs triaged.
 190 <danilos> rockstar, well, we are still very far off zero OOPSes in normal circumstances
 191 <Ursinha> danilos: +1
 192 <rockstar> danilos, this is true, but that's not the issue.
 193 <danilos> rockstar, right, I'd say we have a pretty good situation in translations (rarely more than 10 untriaged bugs), but does it hold for all teams?
 194 <danilos> rockstar, how about launchpad-foundations which is our hold-all bag?
 195 <rockstar> If there's an oops that Ursinha or matsubara sees, they file a bug (they are good at this).  At that point, they should assume the ball has been passed.
 196 <rockstar> danilos, how is launchpad-foundations the "hold-all" bag?  If they have bugs that belong to code, they're hurting foundations and code by not re-assigning them.
 197 <danilos> rockstar, sharing knowledge is useful; when I've seen some OOPSes related to authdb split in translations, I pinged bigjools because I suspected it might bite soyuz as well
 198 <rockstar> danilos, yeah, but that doesn't need to happen in the meeting.
 199 <danilos> rockstar, they have a gazillion bugs which are generic LP problems but not necessarily part of infrastructure, like UI features and such
 200 <sinzui> One problem that I am thinking of was caused by our switch to storm and each application has to make their own fix. This is not a foundations issue, it is an oops that should be targeted to 3 projects
 201 <danilos> sinzui, +1
 202 <rockstar> sinzui, exactly.
 203 <danilos> what I am saying is that we do have problems in some teams that need to be fixed before we can even consider dropping OOPS section from the meeting
 204 <rockstar> Make no mistake.  matsubara and Ursinha are invaluable to Launchpad.  I just don't think they need to duplicate their efforts because I can't triage bugs.
 205 <matsubara> danilos, +1 on the share knowledge usefulness. that's why I think this meeting (and the oops section) is still useful. we bring everyone together and talk about those issues. ideally this would all happen in the bug tracker, but reality is that sometimes some bugs end up falling through the cracks.
 206 <Ursinha> matsubara: +1
 207 <matsubara> and that's where Ursinha and I could help by nagging you guys endlessly :-)
 208 <rockstar> So let's keep the oops section, but not bring up untriaged bugs.
 209 <Ursinha> yes hehe
 210 <danilos> rockstar, I agree, but we first need to make sure "you" are triaging your bugs; care to do the evaluation of all LP subprojects and come back with data that either confirms or rejects my feeling of what it is?
 211 <rockstar> danilos, should QA have to babysit us to triage our bugs?  That doesn't seem fair.
 212 <Ursinha> rockstar: I don't know if that's ideal now; sometimes we dig old triaged bugs (or partially triaged) that need to be taken care now
 213 <danilos> rockstar, no, they shouldn't; can you guarantee all teams will have a very low number of untriaged bugs by next Thursday?
 214 <danilos> Ursinha, rockstar is not complaining about those
 215 <danilos> or at least not strongly :)
 216 <rockstar> danilos, I can't guarantee that, but maybe it's something we should start pushing harder.
 217 <Ursinha> danilos: I'm saying that mentioning bugs isn't useless, triaged or untriaged, because afaik me and matsubara only bring up that ones that need to be taken care of
 218 <danilos> rockstar, what they do today is to raise importance of OOPS bugs in this meeting: sometimes they'll be untriaged, sometimes triaged
 219 <Ursinha> danilos: agreeing with you
 220 <danilos> rockstar, I agree we should start pushing harder, let's have untriaged bugs part of the OOPS section until we feel confident we've done a good push
 221 <matsubara> danilos, so you're asking some kinda of count of untriaged bugs per team in the oops section? we could do that easily, I think.
 222 <danilos> rockstar, we have 218 'New' bugs on launchpad-project today
 223 <rockstar> Alright.  It's obvious that we're at an impass.  I'll bring up my personal feelings with my line manager, and see what comes of it.
 224 <matsubara> danilos, or bring up those untriaged, but until we get those to manageable number, bringing up all of them during the meeting will be difficult and take a lot of time 
 225 <rockstar> I guess leave the meeting as it is.
 226 <danilos> rockstar, now I have to look up 'impass' in a dictionary :)
 227 <rockstar> danilos, deadlock.
 228 <danilos> rockstar, ok, thanks
 229 * bigjools has quit (Remote host closed the connection)
 230 <sinzui> suck to be soyuz it seems
 231 <danilos> rockstar, matsubara: having a per-team count of untriaged bugs would be a good start
 232 <sinzui> and I see one of the new bugs claims to have a patch!
 233 <matsubara> [action] matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section 
 234 <MootBot> ACTION received:  matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
 235 <sinzui> launchpad-foundations has a bug with a patch. I think they are hoping that jml will take the bug away from them
 236 <jml> what
 237 <jml> (gotta ask me first)
 238 * mrevell (~matthew@canonical/launchpad/mrevell) has joined #launchpad-meeting
 239 <sinzui> jml: https://bugs.edge.launchpad.net/launchpad-foundations/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_supervisor=&fie
 240 * sinzui hates bug urls
 241 <danilos> rockstar, btw, I'd be very happy if you get this moving in any way you can, so thanks for raising it!
 242 <danilos> sinzui, we should integrate shorturl service into LP and generate/reuse those URLs with every bugs page :)
 243 <sinzui> danilos: what? implement saved searches for users and teams?
 244 <danilos> sinzui, kind of :)
 245 <danilos> anyway, matsubara, I believe the topic is settled
 246 <sinzui> We certainly would get a lot of cheers from users and even OEM
 247 <matsubara> danilos, thanks for the feedback on this. you too rockstar and sinzui 
 248 <matsubara> anything else before I close?
 249 * bigjools (~quassel@canonical/launchpad/bigjools) has joined #launchpad-meeting
 250 <jml> sinzui, me too
 251 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 252 <jml> sinzui, anyway, I'm busy and not really paying attention. If you want me to do something, please email me something asking about it :)
 253 <matsubara> #endmeeting