1 <MootBot> Meeting started at 10:00. The chair is matsubara.
   2 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   3 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   4 <matsubara> [TOPIC] Roll Call 
   5 <MootBot> New Topic:  Roll Call
   6 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   7 * flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
   8 <flacoste> me
   9 <matsubara> hi francis
  10 <Ursinha> me
  11 <matsubara> rockstar, bigjools, henninge intellectronica sinzui: hi
  12 <sinzui> me
  13 <bigjools> meeee
  14 <intellectronica> me
  15 <henninge> me
  16 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  17 <matsubara> hi stub 
  18 <stub> yo
  19 <matsubara> herb: 
  20 <matsubara> ok, herb can join in later. let's move on
  21 <matsubara> [TOPIC] Agenda 
  22 <MootBot> New Topic:  Agenda
  23 <matsubara>  * Actions from last meeting
  24 <matsubara>  * Oops report & Critical Bugs 
  25 <matsubara>  * Operations report (mthaddon/herb/spm)
  26 <matsubara>  * DBA report (stub)
  27 <matsubara> [TOPIC] * Actions from last meeting
  28 <MootBot> New Topic:  * Actions from last meeting
  29 <matsubara>     * sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
  30 <matsubara>     * ursinha to file a bug about "appserver isn't recovering like it should causing too many oopses"
  31 <matsubara>       * filed bug 360846
  32 <matsubara>     * intellectronica to talk to gmb about bug 269538
  33 <matsubara> ok, Ursinha done hers
  34 <sinzui> wow I do suck
  35 <matsubara> [action] sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
  36 <MootBot> ACTION received:  sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
  37 <sinzui> matsubara: I will do this email IMMEDIATELY after this meeting.
  38 <matsubara> thanks sinzui 
  39 * sinzui starts the subect line now
  40 <intellectronica> matsubara: sorry, i didn't. will try to do that later today
  41 <matsubara> intellectronica: cool. thanks. shall I keep the action item for the next meeting?
  42 <rockstar> me
  43 <intellectronica> matsubara: yes, please
  44 <matsubara> [action] intellectronica to talk to gmb about bug 269538
  45 <MootBot> ACTION received:  intellectronica to talk to gmb about bug 269538
  46 <matsubara> ok, let's move on. thanks sinzui and intellectronica 
  47 <matsubara> hi rockstar 
  48 <rockstar> hi
  49 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  50 <MootBot> New Topic:  * Oops report & Critical Bugs
  51 <matsubara> Ursinha: take the stage please
  52 <Ursinha> I have only one bug for foundations, bug 357307
  53 * Ursinha kicks the bot
  54 <Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/357307
  55 <Ursinha> flacoste, ^
  56 <flacoste> yep, looking at this now
  57 <flacoste> Ursinha: what's the priority on this?
  58 <matsubara> I have one for sinzui: bug 358332
  59 <flacoste> sinzui: do you think salgado could look into that token bug?
  60 <matsubara> curtis, do you agree with the importance I set for https://bugs.edge.launchpad.net/launchpad-registry/+bug/358332 ?
  61 <Ursinha> flacoste, we're had 25 yesterday on lpnet
  62 <Ursinha> about this average a day, oopses count
  63 <sinzui> Well I do not agree with Medium, I do not ever use that status
  64 <matsubara> sinzui: can you reset the importance to a more meaningful value then? 
  65 <sinzui> flacoste: Would you like to take salgado to your team then ;)
  66 <Ursinha> haha
  67 <flacoste> sinzui: you know my opinion on that one :-)
  68 <sinzui> matsubara: Since I have committed to working on series issues, I will try to land a fix for bug 358332 this release
  69 <flacoste> sinzui: and I'll take that as a yes
  70 <matsubara> thank you sinzui 
  71 <sinzui> Yes, I will ask salgado to look at the bug 357307
  72 <matsubara> [action] sinzui to take bug 358332
  73 <MootBot> ACTION received:  sinzui to take bug 358332
  74 <matsubara> [action] sinzui to ask salgado to fix bug 357307
  75 <MootBot> ACTION received:  sinzui to ask salgado to fix bug 357307
  76 <matsubara> All critical bugs are fix committed
  77 <matsubara> so, I think that's all. anything else Ursinha ?
  78 <Ursinha> no matsubara
  79 <matsubara> ok, thanks everyone
  80 <Ursinha> thanks all
  81 <matsubara> I'm going to skip herb's section since he's not here yet.
  82 <mthaddon> matsubara: I'm here
  83 <matsubara> oh
  84 <matsubara> hi mthaddon 
  85 <mthaddon> matsubara: herb's off
  86 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
  87 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
  88 <mthaddon>  - Preparations for new rollout procedure - need to confirm a fair amount of stuff with stub before the next rollout
  89 <mthaddon>  - PostgreSQL upgrade yesterday meant some downtime
  90 <mthaddon>  - Need to decide on whether we're switching the master to wildcherry and if so when
  91 <mthaddon> I think that's it unless there are any questions
  92 <intellectronica> mthaddon: i have a question
  93 <matsubara> what's this new rollout procedure?
  94 <mthaddon> matsubara: SSO related
  95 <mthaddon> intellectronica: sure
  96 <intellectronica> mthaddon: yesterday morning (UTC) forster needed a restart because it was overwhelmed. Spads helped me with that. do you know what happened?
  97 <mthaddon> matsubara: involves splitting out a slave from as standalone so we can continue to serve SSO
  98 <mthaddon> intellectronica: I don't - we've had some issues with forster over the past few weeks
  99 <matsubara> mthaddon: cool.
 100 <mthaddon> intellectronica: i.e. this isn't the first time it's needed a kicking - I'll see if I can look into it a little more
 101 <intellectronica> mthaddon: ok, as long as you're not surprised. sorry i forgot to mention this yesterday
 102 <matsubara> anything else for mthaddon?
 103 <mthaddon> intellectronica: we're moving the incoming email parsing script off there, so hopefully that'll reduce load a little, although I'd be surprised if it was related to that
 104 <flacoste> mthaddon: any idea on the surge in buildbot failures yesterday?
 105 <mthaddon> flacoste: I was off yesterday so I wasn't aware of that, no
 106 <flacoste> mthaddon: it looks it was restarted, i assume by spm since gary nor i did it
 107 <intellectronica> mthaddon: if it's not the cause, then at least it won't stop working when whatever is the cause starts going wild :)
 108 <flacoste> mthaddon: ok, i'll check with spm later
 109 <mthaddon> intellectronica: absolutely :)
 110 <matsubara> flacoste: maybe it was related to the unexpected downtime we had yesterday
 111 <matsubara> flacoste: all over the DC
 112 <flacoste> matsubara: none of this lives in the DC yet
 113 <matsubara> well, but the code buildbot is pulling does :-)
 114 <matsubara> anyway, spm can clarify that later.
 115 <matsubara> thanks everyone
 116 <matsubara> [TOPIC] * DBA report (stub)
 117 <MootBot> New Topic:  * DBA report (stub)
 118 <stub> I have been on public holidays or ill for the last week, but things still seem to be running which is nice.
 119 <stub> We got a replication lag spike the other night - all the systems coped well while it was cleared. Possibly the rosetta imports being busier than usual. Possibly the rosetta imports where just delayed by something else. I don't think anyone has traced what processes where running at that time yet from the script activity records or the output on launchpad-errors.
 120 <stub> We are going to have to be more agressive at blocking scripts when the system is under load though. I might be able to retrofit this to all our existing scripts by blocking after a commit or rollback if the system is loaded using similar rules to the garbage collector (no transactions open longer than 30 mins and replication lag < 30 seconds).
 121 <stub> Yes, we should switch the master to wildcherry before the next rollout or during the next rollout if we don't mind a few minutes downtime on the SSO server.
 122 <stub> I need to go over rollout procedures with the losas once I know how much of read-only-launchpad lands this cycle.
 123 <stub> EOM
 124 <mthaddon> stub: when do you expect to know how much of read-only-launchpad lands this cycle?
 125 <stub> I don't know yet ;) What would be a good deadline?
 126 <mthaddon> stub: I think we'd need to discuss things at the latest by the end of week 3, so sometime before then?
 127 <mthaddon> in terms of rollout stuff, I mean
 128 <stub> Sure. I was thinking early next week to fix the plan so that works.
 129 <matsubara> anything else for stub?
 130 <matsubara> or anything else before I close?
 131 <matsubara> thanks stub 
 132 <mthaddon> stub: cool, thx
 133 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 134 <matsubara> #endmeeting 
 135 <MootBot> Meeting finished at 10:29.