## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<bigjools> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<rockstar> ni!
<bac> me
<stub> me
<gary_poster> me
<EdwinGrubbs> me, sitting in for sinzui
<matsubara> Ursinha, Chex: hi
* flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
<Ursinha> me
<matsubara> danilos, are you joining us?
* Chex is here
<Ursinha> matsubara, flaky internet
<matsubara> allenap, hi
<danilos> me
<danilos> matsubara: yes
<allenap> me
<matsubara> danilos, shall I make you the default translations person for the LP prod meeting?
<mbarnett> me?
<gary_poster> you!
<matsubara> hi mbarnett 
<matsubara> welcome :-)
<mbarnett>  :)
<matsubara> ok, everyone is here
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>     * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<matsubara>     * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<danilos> matsubara: yes
<matsubara> I still suck, haven't done that one.
<matsubara> [action]  * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<MootBot> ACTION received:   * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<matsubara> EdwinGrubbs, I'll keep the action item for barry in the list since 3.0 crazyness is not over yet :-)
<matsubara> [action] * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<MootBot> ACTION received:  * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<matsubara> [action] matsubara to update translations qa contact on meeting agenda page
<MootBot> ACTION received:  matsubara to update translations qa contact on meeting agenda page
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> Ursinha, go ahead
<matsubara> hmm Ursinha was having connectivity problems before the meeting
<danilos> ok, so, let me take over for what I can
<danilos> bac is doing a great job with managing release of 3.0 and any critical bugs
<danilos> what we've seen is already listed on https://dev.launchpad.net/CurrentRolloutBlockers
<bac> thanks.  getting lots of good help.
<danilos> we have noticed increased timeouts on production DBs, due to postgres restart
<danilos> for more details, Ursinha will have to fill us in
<matsubara> we had a bunch of disconnection errors as well
<danilos> matsubara: around the time of rollout or? (because that'd be expected if so)
* Ursinha has quit (Read error: 113 (No route to host))
<matsubara> danilos, not sure, Ursinha just sms'ed me and asked me to bring it up
<matsubara> but I take the increase on those are due to the rollout, yes
<matsubara> so, Ursinha and I will keep looking for the new oopses that happened since the rollout
<danilos> matsubara: I suggest we get more detailed input from Ursula when she shows up back online, but go on with the meeting for now; for what we know, all critical issues are being taken care of, and we are hoping for a re-rollout tonight
<matsubara> and let each affected team about them and add them to the CRB page
<danilos> matsubara: thanks matsubara, that'd be great
<matsubara> we had some script failures as well related to the rollout
<matsubara> and another one unrelated which bigjools already replied to
<matsubara> (which is the packagediffs thing)
<matsubara> we have 6 critical bugs
<bigjools> fixed in the rollout
<matsubara> 2 fix committed, 3 in progress and one triaged
<matsubara> danilos, the one triaged is in rosetta. what's up? are you expecting to land code for that one for the second rollout?
<matsubara> bac, do you have a time for the second rollout?
<danilos> matsubara: it's in progress
<matsubara> thanks bigjools 
<matsubara> danilos, I mean bug 435891
<ubottu> Launchpad bug 435891 in rosetta "recent update broke the urls used in launchpad integration" [Critical,Triaged] https://launchpad.net/bugs/435891
<danilos> matsubara: it's in progress
<bac> matsubara: we anticipate a second roll out.  perhaps later today, though no decision has been made yet.
<matsubara> danilos, it's not what LP says :-)
<danilos> matsubara: it's what I say :)
<danilos> bug 435891 is in progress :)
<matsubara> danilos, I trust you to beat LP and make it behave :-)
<matsubara> thanks danilos 
<danilos> the other in-progress one is almost landed (in pqm)
<flacoste> matsubara: Disconnection errors
<flacoste> about them
<flacoste> do you know if they happened before or after the roll-out?
<flacoste> if they happened before/during the roll-out they are expected
<flacoste> we had a bug, now fixed, that logged DisconnectionError as regular OOPS
<bac> are you in the production meeting?
<flacoste> where it should be logged as a soft oops
<danilos> bac: yes, this is the production meeting
<bac> sorry, wrong window
<flacoste> so DisconnectionError logged after the roll-out are a problem
<matsubara> flacoste, after the meeting I'll generate a new summary to find that out.
<flacoste> ones before or during it (when running with unfixed code) is not a problem
<matsubara> [action] matsubara to generate new oops summary including oopses only for after the rollout
<MootBot> ACTION received:  matsubara to generate new oops summary including oopses only for after the rollout
<flacoste> matsubara, bac: we can determine the second roll-out time after we have that report
<matsubara> I'll defer the answer to bac
<bac> matsubara: i was just made aware of bug 435628 that the bugs team is working on for a re-roll
<ubottu> Launchpad bug 435628 in malone "Attempting to file a bug on an Ubuntu source packages OOPSes when bug filing is disabled" [High,Triaged] https://launchpad.net/bugs/435628
<stub> barry was going to look at annotating those oopses so we can tell if they where DisconnectionErrors returned to users, or just part of the normal reconnection workflow.
<bac> matsubara: i agree with flacoste that we can make a decision after getting your report
<matsubara> all right
<stub> erm... gary... not barry
<gary_poster> :-)
<matsubara> heh
<matsubara> ok, I think that's all for this section
<matsubara> let's move on
<matsubara> thanks everyone!
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Chex> HI everyone, brief LOSA report this week.
<Chex> - LP 3.0 rollout: The rollout dominated Launchpad work this week. Briefly, things went fairly well, we
<Chex> observed a number of items in the rollout process that are being addressed.  We have emailed out the
<Chex> full report today.
<Chex> And that it for us, unless anyone has questions?
<gary_poster> three cheers for losas. :-)
<matsubara> thanks Chex
<mthaddon> gary_poster: I think you mean three *beers* for each losa...
<gary_poster> lol, yeah, probably more appreciated
<mthaddon> :)
* stub raises a glass and says 'cheers'
* Chex drinks to that
<danilos> so, drunken DB report comes next? :)
<matsubara> let's go to the bar, err, move on
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
* stub is now known as drunken_master
<danilos> :)
<drunken_master> Database update seems to have gone smoothly apart from Bug #435674 being discovered. This will have caused a short hang of the SSO servers, but it was probably quick and hopefully nobody noticed.
<drunken_master> After the update, I needed to do some data migration. That increased replication lag, putting extra load on the master db. All done now and load is back to normal. This will have contributed to timeouts, so you may wants to attribute timeouts before 1200 UTC to this.
<drunken_master> The appservers where observed in the wild basing their decision to use the slave database on how lagged that particular slave is, not how lagged the cluster is as a whole. This means that building new slave databases will no longer stop the appservers going into master-only mode.
<rockstar> :)
<ubottu> Launchpad bug 435674 in launchpad-foundations "fti.py wants to lock all replication sets" [High,Triaged] https://launchpad.net/bugs/435674
<drunken_master> oot.
<gary_poster> lol
* drunken_master is now known as stub
<mbarnett> heh
<danilos> thanks for the update drunken_master, always appreciated
<matsubara> thanks stub 
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no proposed items
* Ursinha (n=ursula@canonical/launchpad/ursinha) has joined #launchpad-meeting
<matsubara> anything else before I close?
<matsubara> hi Ursinha 
<matsubara> just in time :-)
<Ursinha> I just want to say sorry
<Ursinha> and congrats to all lp team because LP 3.0 rocks
<Ursinha> :)
<matsubara> indeed!
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:27.