## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<Ursinha> me!
<allenap> me
<henninge> me
<al-maisan> me
<matsubara> Chex, rockstar: hi
<matsubara> bac, can you stand in for sinzui?
<Chex> matsubara: hello
<bac> matsubara:  gladly
<bac> me
<Ursinha> :)
<matsubara> thanks bac 
<jtv> me
<Ursinha> oh, a special guest :P
<matsubara> I'm standing in for gary
<matsubara> hi jtv 
<jtv> hi
<matsubara> we have 3/4 of the translations team around
<matsubara> pretty cool
<jtv> with all the oopses going on, we thought we'd look in just in case
* Ursinha hi-fives henninge and jtv 
* jtv hi-fives back
<matsubara> ok, let's get this started
<Ursinha> yay
<matsubara> [TOPIC] Agenda 
* henninge ducks
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * add brian to the list of attendees in the MeetingAgenda page
<matsubara>     * done
<matsubara> * allenap to dig the master bug of OOPS-1474EA771
<matsubara>     * done. it's #508302
<matsubara> * rockstar to take a look in OOPS-1480CMP1
<matsubara> * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174
<matsubara> * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
<matsubara> * matsubara to talk to salgado about More non-informational disconnectionerrors https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489J147
<matsubara> * matsubara to talk to stub or gary about InternalError after ther rollout https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489C1094
<matsubara>     * https://launchpad.net/bugs/514267
<matsubara> * matsubara to fix bug link on OOPS-1474EA771 to point to bug 508302
<matsubara>     * done
<matsubara> * sinzui to investigate failure on send-person-notifications and reply to the list with his findings
<matsubara>     * https://bugs.edge.launchpad.net/launchpad-registry/+bug/514149
<matsubara> * rockstar to raise the importance of ensuring sufficient monitoring of this part (i.e. branch updates emails failing to be delivered) of code-hosting by thumper
<matsubara> * Gary talk about "too many different kinds of moving parts" in our architecture. If we have fewer moving parts then we can institute more uniform nagios-like-checks.
<matsubara> * matsubara to update oops-tools to know about the new lp appserver
<matsubara>     * done.
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489J147
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489C1094
<Ursinha> oh dear
<ubottu> Ubuntu bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Triaged]
<matsubara> hmm ok, rockstar is not around
<ubottu> Launchpad bug 508302 in malone "NotImplementedError OOPS when reporting a bug" [High,Fix committed] https://launchpad.net/bugs/508302
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<ubottu> Ubuntu bug 514149 in launchpad-registry "send-person-notifications.py is broken" [High,Fix released]
<matsubara> I haven't talked to leonard nor emailed team about those oopses
<matsubara> [action] * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174 and * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
<MootBot> ACTION received:  * matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174 and * matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
<matsubara> [action] * rockstar to take a look in OOPS-1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<MootBot> ACTION received:  * rockstar to take a look in OOPS-1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<matsubara> I guess that's it
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> okay, so
<Ursinha> the alarming bug is bug 514267
<ubottu> Launchpad bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Triaged] https://launchpad.net/bugs/514267
<Ursinha> apparently it's the reason of about 6k InternalError oopses only today in lpnet
<matsubara> stub is on it, discussion ongoing in the internal channel
<Ursinha> matsubara: okay, notes in that bug are welcome :)
<matsubara> feel free to add them Ursinha 
<Ursinha> matsubara: thanks
<Ursinha> matsubara: I don't know what's being discussed :)
<Ursinha> anyway, I guess this is most critical thing
<Ursinha> we have six critical bugs, two in progress
<matsubara> no other oopses worth mentioning?
<Ursinha> matsubara: not that already have bugs/are being worked on, afaik
<matsubara> ok
<Ursinha> matsubara: bug 511567 is triaged and marked to 10.01
<ubottu> Launchpad bug 511567 in launchpad-foundations "Can't remove authorised oauth tokens" [Critical,Triaged] https://launchpad.net/bugs/511567
<Ursinha> is that really critical?
<matsubara> 10.01 should be closed by now, so I guess it should be 10.02
<Ursinha> matsubara: right, but is that really critical?
<matsubara> don't know I asked in the bug report to Curtis who set it as critical
<Ursinha> bug 514461 is code, in progress, have to check with rockstar how's that
<ubottu> Launchpad bug 514461 in launchpad-code "scan_branches.py screams and no one is listening" [Critical,In progress] https://launchpad.net/bugs/514461
<Ursinha> [action] Ursinha to talk to rockstar about bug 514461
* henninge_ (~henning@canonical/launchpad/henninge) has joined #launchpad-meeting
<Ursinha> [action] Ursinha to talk to mwhudson about bug 513412
<ubottu> Bug 513412 on http://launchpad.net/bugs/513412 is private
<Ursinha> and, finally, bug 515473 is assigned to curtis and in progress
<ubottu> Launchpad bug 515473 in soyuz "ppa description contains smal links" [Critical,In progress] https://launchpad.net/bugs/515473
<Ursinha> bac: do you know the progress on that one?
<bac> i do not
<Ursinha> bac: can you ask him, if possible, please? :)
<bac> looking into it
<Ursinha> bac: thanks
<Ursinha> bug 508025 is only waiting for a test commit so it can be closed
<ubottu> Launchpad bug 508025 in lpbuildbot "Need production-devel builder" [Critical,In progress] https://launchpad.net/bugs/508025
<Ursinha> end of critical bugs
<Ursinha> now for something completely different
* henninge has quit (Ping timeout: 272 seconds)
<Ursinha> script process-pending-packagediffs has been failing lately, is that soyuz? al-maisan?
<al-maisan> al-maisan: I have a branch that fixes these failure
* henninge_ is now known as henninge
<al-maisan> it has been reviewed and I am revising it in accordance with the review comments
<Ursinha> that's great al-maisan, I'll reply in the failure mail
<Ursinha> al-maisan: thanks for the info
<Ursinha> matsubara: you can move on
<al-maisan> you are welcome
<matsubara> thanks Ursinha 
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<rockstar> me
<Ursinha> rockstar: good morning :)
<Chex> hi everyone
<rockstar> Ursinha, :)
<Chex> here is the LOSA report for this week:
<Chex> - LP Outage Today @1400UTC: LP is back running again after a outage; there was an issue with the new
<Chex>         DB replication lag check not working properly. We are working to see why this happened and
<Chex>         resolve.
<Chex> - LP incidents of note:
<Chex>         ; LP Cherry-picks: rolled out 8940 to Lpnet on 29-Jan
<Chex> Does anyone have any comments or questions?
<matsubara> thanks Chex 
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> I'll ask stub to send the report to the list
<matsubara> [action] matsubara to ask stub to send the dba report to the list
<MootBot> ACTION received:  matsubara to ask stub to send the dba report to the list
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no proposed items
<matsubara> ok, since we have some time
<matsubara> * rockstar to take a look in OOPS-1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<matsubara> rockstar, that one is in the actions from last meeting topic for quite some week now
<matsubara> did you have time to look into that?
<rockstar> matsubara, I'll take a look.
<Ursinha> thanks rockstar
<matsubara> rockstar, I thought you were looking and would report back here...
<matsubara> shall I close the meeting then?
<rockstar> matsubara, that's the first I've seen that oops.  I'm not sure how I missed it from other meetings.
<matsubara> by missing the meeting? :-)
<matsubara> rockstar, anyway, thanks for looking into that. if it helps I'll file a place holder bug and you can fill in the details once you know more about it
<matsubara> thanks everyone
<rockstar> matsubara, sure.
<Ursinha> thanks matsubara
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:30.