1 <MootBot> Meeting started at 10:00. The chair is matsubara.
   2 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   3 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   4 <matsubara> [TOPIC] Roll Call 
   5 <MootBot> New Topic:  Roll Call
   6 <bigjools> me
   7 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   8 <rockstar> ni!
   9 <bac> me
  10 <stub> me
  11 <gary_poster> me
  12 <EdwinGrubbs> me, sitting in for sinzui
  13 <matsubara> Ursinha, Chex: hi
  14 * flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
  15 <Ursinha> me
  16 <matsubara> danilos, are you joining us?
  17 * Chex is here
  18 <Ursinha> matsubara, flaky internet
  19 <matsubara> allenap, hi
  20 <danilos> me
  21 <danilos> matsubara: yes
  22 <allenap> me
  23 <matsubara> danilos, shall I make you the default translations person for the LP prod meeting?
  24 <mbarnett> me?
  25 <gary_poster> you!
  26 <matsubara> hi mbarnett 
  27 <matsubara> welcome :-)
  28 <mbarnett>  :)
  29 <matsubara> ok, everyone is here
  30 <matsubara> [TOPIC] Agenda 
  31 <MootBot> New Topic:  Agenda
  32 <matsubara>  * Actions from last meeting
  33 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  34 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  35 <matsubara>  * DBA report (stub)
  36 <matsubara>  * Proposed items
  37 <matsubara> [TOPIC] * Actions from last meeting
  38 <MootBot> New Topic:  * Actions from last meeting
  39 <matsubara>     * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  40 <matsubara>     * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  41 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  42 <danilos> matsubara: yes
  43 <matsubara> I still suck, haven't done that one.
  44 <matsubara> [action]  * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  45 <MootBot> ACTION received:   * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  46 <matsubara> EdwinGrubbs, I'll keep the action item for barry in the list since 3.0 crazyness is not over yet :-)
  47 <matsubara> [action] * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  48 <MootBot> ACTION received:  * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  49 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  50 <matsubara> [action] matsubara to update translations qa contact on meeting agenda page
  51 <MootBot> ACTION received:  matsubara to update translations qa contact on meeting agenda page
  52 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  53 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  54 <matsubara> Ursinha, go ahead
  55 <matsubara> hmm Ursinha was having connectivity problems before the meeting
  56 <danilos> ok, so, let me take over for what I can
  57 <danilos> bac is doing a great job with managing release of 3.0 and any critical bugs
  58 <danilos> what we've seen is already listed on https://dev.launchpad.net/CurrentRolloutBlockers
  59 <bac> thanks.  getting lots of good help.
  60 <danilos> we have noticed increased timeouts on production DBs, due to postgres restart
  61 <danilos> for more details, Ursinha will have to fill us in
  62 <matsubara> we had a bunch of disconnection errors as well
  63 <danilos> matsubara: around the time of rollout or? (because that'd be expected if so)
  64 * Ursinha has quit (Read error: 113 (No route to host))
  65 <matsubara> danilos, not sure, Ursinha just sms'ed me and asked me to bring it up
  66 <matsubara> but I take the increase on those are due to the rollout, yes
  67 <matsubara> so, Ursinha and I will keep looking for the new oopses that happened since the rollout
  68 <danilos> matsubara: I suggest we get more detailed input from Ursula when she shows up back online, but go on with the meeting for now; for what we know, all critical issues are being taken care of, and we are hoping for a re-rollout tonight
  69 <matsubara> and let each affected team about them and add them to the CRB page
  70 <danilos> matsubara: thanks matsubara, that'd be great
  71 <matsubara> we had some script failures as well related to the rollout
  72 <matsubara> and another one unrelated which bigjools already replied to
  73 <matsubara> (which is the packagediffs thing)
  74 <matsubara> we have 6 critical bugs
  75 <bigjools> fixed in the rollout
  76 <matsubara> 2 fix committed, 3 in progress and one triaged
  77 <matsubara> danilos, the one triaged is in rosetta. what's up? are you expecting to land code for that one for the second rollout?
  78 <matsubara> bac, do you have a time for the second rollout?
  79 <danilos> matsubara: it's in progress
  80 <matsubara> thanks bigjools 
  81 <matsubara> danilos, I mean bug 435891
  82 <ubottu> Launchpad bug 435891 in rosetta "recent update broke the urls used in launchpad integration" [Critical,Triaged] https://launchpad.net/bugs/435891
  83 <danilos> matsubara: it's in progress
  84 <bac> matsubara: we anticipate a second roll out.  perhaps later today, though no decision has been made yet.
  85 <matsubara> danilos, it's not what LP says :-)
  86 <danilos> matsubara: it's what I say :)
  87 <danilos> bug 435891 is in progress :)
  88 <matsubara> danilos, I trust you to beat LP and make it behave :-)
  89 <matsubara> thanks danilos 
  90 <danilos> the other in-progress one is almost landed (in pqm)
  91 <flacoste> matsubara: Disconnection errors
  92 <flacoste> about them
  93 <flacoste> do you know if they happened before or after the roll-out?
  94 <flacoste> if they happened before/during the roll-out they are expected
  95 <flacoste> we had a bug, now fixed, that logged DisconnectionError as regular OOPS
  96 <bac> are you in the production meeting?
  97 <flacoste> where it should be logged as a soft oops
  98 <danilos> bac: yes, this is the production meeting
  99 <bac> sorry, wrong window
 100 <flacoste> so DisconnectionError logged after the roll-out are a problem
 101 <matsubara> flacoste, after the meeting I'll generate a new summary to find that out.
 102 <flacoste> ones before or during it (when running with unfixed code) is not a problem
 103 <matsubara> [action] matsubara to generate new oops summary including oopses only for after the rollout
 104 <MootBot> ACTION received:  matsubara to generate new oops summary including oopses only for after the rollout
 105 <flacoste> matsubara, bac: we can determine the second roll-out time after we have that report
 106 <matsubara> I'll defer the answer to bac
 107 <bac> matsubara: i was just made aware of bug 435628 that the bugs team is working on for a re-roll
 108 <ubottu> Launchpad bug 435628 in malone "Attempting to file a bug on an Ubuntu source packages OOPSes when bug filing is disabled" [High,Triaged] https://launchpad.net/bugs/435628
 109 <stub> barry was going to look at annotating those oopses so we can tell if they where DisconnectionErrors returned to users, or just part of the normal reconnection workflow.
 110 <bac> matsubara: i agree with flacoste that we can make a decision after getting your report
 111 <matsubara> all right
 112 <stub> erm... gary... not barry
 113 <gary_poster> :-)
 114 <matsubara> heh
 115 <matsubara> ok, I think that's all for this section
 116 <matsubara> let's move on
 117 <matsubara> thanks everyone!
 118 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 119 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 120 <Chex> HI everyone, brief LOSA report this week.
 121 <Chex> - LP 3.0 rollout: The rollout dominated Launchpad work this week. Briefly, things went fairly well, we
 122 <Chex> observed a number of items in the rollout process that are being addressed.  We have emailed out the
 123 <Chex> full report today.
 124 <Chex> And that it for us, unless anyone has questions?
 125 <gary_poster> three cheers for losas. :-)
 126 <matsubara> thanks Chex
 127 <mthaddon> gary_poster: I think you mean three *beers* for each losa...
 128 <gary_poster> lol, yeah, probably more appreciated
 129 <mthaddon> :)
 130 * stub raises a glass and says 'cheers'
 131 * Chex drinks to that
 132 <danilos> so, drunken DB report comes next? :)
 133 <matsubara> let's go to the bar, err, move on
 134 <matsubara> [TOPIC] * DBA report (stub)
 135 <MootBot> New Topic:  * DBA report (stub)
 136 * stub is now known as drunken_master
 137 <danilos> :)
 138 <drunken_master> Database update seems to have gone smoothly apart from Bug #435674 being discovered. This will have caused a short hang of the SSO servers, but it was probably quick and hopefully nobody noticed.
 139 <drunken_master> After the update, I needed to do some data migration. That increased replication lag, putting extra load on the master db. All done now and load is back to normal. This will have contributed to timeouts, so you may wants to attribute timeouts before 1200 UTC to this.
 140 <drunken_master> The appservers where observed in the wild basing their decision to use the slave database on how lagged that particular slave is, not how lagged the cluster is as a whole. This means that building new slave databases will no longer stop the appservers going into master-only mode.
 141 <rockstar> :)
 142 <ubottu> Launchpad bug 435674 in launchpad-foundations "fti.py wants to lock all replication sets" [High,Triaged] https://launchpad.net/bugs/435674
 143 <drunken_master> oot.
 144 <gary_poster> lol
 145 * drunken_master is now known as stub
 146 <mbarnett> heh
 147 <danilos> thanks for the update drunken_master, always appreciated
 148 <matsubara> thanks stub 
 149 <matsubara> [TOPIC] * Proposed items
 150 <MootBot> New Topic:  * Proposed items
 151 <matsubara> no proposed items
 152 * Ursinha (n=ursula@canonical/launchpad/ursinha) has joined #launchpad-meeting
 153 <matsubara> anything else before I close?
 154 <matsubara> hi Ursinha 
 155 <matsubara> just in time :-)
 156 <Ursinha> I just want to say sorry
 157 <Ursinha> and congrats to all lp team because LP 3.0 rocks
 158 <Ursinha> :)
 159 <matsubara> indeed!
 160 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 161 <matsubara> #endmeeting 
 162 <MootBot> Meeting finished at 10:27.