1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   8 <Ursinha> me!
   9 <allenap> me
  10 <henninge> me
  11 <al-maisan> me
  12 <matsubara> Chex, rockstar: hi
  13 <matsubara> bac, can you stand in for sinzui?
  14 <Chex> matsubara: hello
  15 <bac> matsubara:  gladly
  16 <bac> me
  17 <Ursinha> :)
  18 <matsubara> thanks bac 
  19 <jtv> me
  20 <Ursinha> oh, a special guest :P
  21 <matsubara> I'm standing in for gary
  22 <matsubara> hi jtv 
  23 <jtv> hi
  24 <matsubara> we have 3/4 of the translations team around
  25 <matsubara> pretty cool
  26 <jtv> with all the oopses going on, we thought we'd look in just in case
  27 * Ursinha hi-fives henninge and jtv 
  28 * jtv hi-fives back
  29 <matsubara> ok, let's get this started
  30 <Ursinha> yay
  31 <matsubara> [TOPIC] Agenda 
  32 * henninge ducks
  33 <MootBot> New Topic:  Agenda
  34 <matsubara>  * Actions from last meeting
  35 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  36 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  37 <matsubara>  * DBA report (stub)
  38 <matsubara>  * Proposed items
  39 <matsubara> [TOPIC] * Actions from last meeting
  40 <MootBot> New Topic:  * Actions from last meeting
  41 <matsubara> * add brian to the list of attendees in the MeetingAgenda page
  42 <matsubara>     * done
  43 <matsubara> * allenap to dig the master bug of OOPS-1474EA771
  44 <matsubara>     * done. it's #508302
  45 <matsubara> * rockstar to take a look in OOPS-1480CMP1
  46 <matsubara> * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174
  47 <matsubara> * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
  48 <matsubara> * matsubara to talk to salgado about More non-informational disconnectionerrors https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489J147
  49 <matsubara> * matsubara to talk to stub or gary about InternalError after ther rollout https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489C1094
  50 <matsubara>     * https://launchpad.net/bugs/514267
  51 <matsubara> * matsubara to fix bug link on OOPS-1474EA771 to point to bug 508302
  52 <matsubara>     * done
  53 <matsubara> * sinzui to investigate failure on send-person-notifications and reply to the list with his findings
  54 <matsubara>     * https://bugs.edge.launchpad.net/launchpad-registry/+bug/514149
  55 <matsubara> * rockstar to raise the importance of ensuring sufficient monitoring of this part (i.e. branch updates emails failing to be delivered) of code-hosting by thumper
  56 <matsubara> * Gary talk about "too many different kinds of moving parts" in our architecture. If we have fewer moving parts then we can institute more uniform nagios-like-checks.
  57 <matsubara> * matsubara to update oops-tools to know about the new lp appserver
  58 <matsubara>     * done.
  59 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
  60 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
  61 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
  62 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
  63 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489J147
  64 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489C1094
  65 <Ursinha> oh dear
  66 <ubottu> Ubuntu bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Triaged]
  67 <matsubara> hmm ok, rockstar is not around
  68 <ubottu> Launchpad bug 508302 in malone "NotImplementedError OOPS when reporting a bug" [High,Fix committed] https://launchpad.net/bugs/508302
  69 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
  70 <ubottu> Ubuntu bug 514149 in launchpad-registry "send-person-notifications.py is broken" [High,Fix released]
  71 <matsubara> I haven't talked to leonard nor emailed team about those oopses
  72 <matsubara> [action] * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174 and * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
  73 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
  74 <MootBot> ACTION received:  * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174 and * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
  75 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
  76 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
  77 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
  78 <matsubara> [action] * rockstar to take a look in OOPS-1480CMP1
  79 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
  80 <MootBot> ACTION received:  * rockstar to take a look in OOPS-1480CMP1
  81 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
  82 <matsubara> I guess that's it
  83 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  84 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  85 <Ursinha> okay, so
  86 <Ursinha> the alarming bug is bug 514267
  87 <ubottu> Launchpad bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Triaged] https://launchpad.net/bugs/514267
  88 <Ursinha> apparently it's the reason of about 6k InternalError oopses only today in lpnet
  89 <matsubara> stub is on it, discussion ongoing in the internal channel
  90 <Ursinha> matsubara: okay, notes in that bug are welcome :)
  91 <matsubara> feel free to add them Ursinha 
  92 <Ursinha> matsubara: thanks
  93 <Ursinha> matsubara: I don't know what's being discussed :)
  94 <Ursinha> anyway, I guess this is most critical thing
  95 <Ursinha> we have six critical bugs, two in progress
  96 <matsubara> no other oopses worth mentioning?
  97 <Ursinha> matsubara: not that already have bugs/are being worked on, afaik
  98 <matsubara> ok
  99 <Ursinha> matsubara: bug 511567 is triaged and marked to 10.01
 100 <ubottu> Launchpad bug 511567 in launchpad-foundations "Can't remove authorised oauth tokens" [Critical,Triaged] https://launchpad.net/bugs/511567
 101 <Ursinha> is that really critical?
 102 <matsubara> 10.01 should be closed by now, so I guess it should be 10.02
 103 <Ursinha> matsubara: right, but is that really critical?
 104 <matsubara> don't know I asked in the bug report to Curtis who set it as critical
 105 <Ursinha> bug 514461 is code, in progress, have to check with rockstar how's that
 106 <ubottu> Launchpad bug 514461 in launchpad-code "scan_branches.py screams and no one is listening" [Critical,In progress] https://launchpad.net/bugs/514461
 107 <Ursinha> [action] Ursinha to talk to rockstar about bug 514461
 108 * henninge_ (~henning@canonical/launchpad/henninge) has joined #launchpad-meeting
 109 <Ursinha> [action] Ursinha to talk to mwhudson about bug 513412
 110 <ubottu> Bug 513412 on http://launchpad.net/bugs/513412 is private
 111 <Ursinha> and, finally, bug 515473 is assigned to curtis and in progress
 112 <ubottu> Launchpad bug 515473 in soyuz "ppa description contains smal links" [Critical,In progress] https://launchpad.net/bugs/515473
 113 <Ursinha> bac: do you know the progress on that one?
 114 <bac> i do not
 115 <Ursinha> bac: can you ask him, if possible, please? :)
 116 <bac> looking into it
 117 <Ursinha> bac: thanks
 118 <Ursinha> bug 508025 is only waiting for a test commit so it can be closed
 119 <ubottu> Launchpad bug 508025 in lpbuildbot "Need production-devel builder" [Critical,In progress] https://launchpad.net/bugs/508025
 120 <Ursinha> end of critical bugs
 121 <Ursinha> now for something completely different
 122 * henninge has quit (Ping timeout: 272 seconds)
 123 <Ursinha> script process-pending-packagediffs has been failing lately, is that soyuz? al-maisan?
 124 <al-maisan> al-maisan: I have a branch that fixes these failure
 125 * henninge_ is now known as henninge
 126 <al-maisan> it has been reviewed and I am revising it in accordance with the review comments
 127 <Ursinha> that's great al-maisan, I'll reply in the failure mail
 128 <Ursinha> al-maisan: thanks for the info
 129 <Ursinha> matsubara: you can move on
 130 <al-maisan> you are welcome
 131 <matsubara> thanks Ursinha 
 132 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 133 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 134 <rockstar> me
 135 <Ursinha> rockstar: good morning :)
 136 <Chex> hi everyone
 137 <rockstar> Ursinha, :)
 138 <Chex> here is the LOSA report for this week:
 139 <Chex> - LP Outage Today @1400UTC: LP is back running again after a outage; there was an issue with the new
 140 <Chex>         DB replication lag check not working properly. We are working to see why this happened and
 141 <Chex>         resolve.
 142 <Chex> - LP incidents of note:
 143 <Chex>         ; LP Cherry-picks: rolled out 8940 to Lpnet on 29-Jan
 144 <Chex> Does anyone have any comments or questions?
 145 <matsubara> thanks Chex 
 146 <matsubara> [TOPIC] * DBA report (stub)
 147 <MootBot> New Topic:  * DBA report (stub)
 148 <matsubara> I'll ask stub to send the report to the list
 149 <matsubara> [action] matsubara to ask stub to send the dba report to the list
 150 <MootBot> ACTION received:  matsubara to ask stub to send the dba report to the list
 151 <matsubara> [TOPIC] * Proposed items
 152 <MootBot> New Topic:  * Proposed items
 153 <matsubara> no proposed items
 154 <matsubara> ok, since we have some time
 155 <matsubara> * rockstar to take a look in OOPS-1480CMP1
 156 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
 157 <matsubara> rockstar, that one is in the actions from last meeting topic for quite some week now
 158 <matsubara> did you have time to look into that?
 159 <rockstar> matsubara, I'll take a look.
 160 <Ursinha> thanks rockstar
 161 <matsubara> rockstar, I thought you were looking and would report back here...
 162 <matsubara> shall I close the meeting then?
 163 <rockstar> matsubara, that's the first I've seen that oops.  I'm not sure how I missed it from other meetings.
 164 <matsubara> by missing the meeting? :-)
 165 <matsubara> rockstar, anyway, thanks for looking into that. if it helps I'll file a place holder bug and you can fill in the details once you know more about it
 166 <matsubara> thanks everyone
 167 <rockstar> matsubara, sure.
 168 <Ursinha> thanks matsubara
 169 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 170 <matsubara> #endmeeting 
 171 <MootBot> Meeting finished at 10:30.