## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
* flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
<flacoste> me
<matsubara> hi francis
<Ursinha> me
<matsubara> rockstar, bigjools, henninge intellectronica sinzui: hi
<sinzui> me
<bigjools> meeee
<intellectronica> me
<henninge> me
* stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
<matsubara> hi stub 
<stub> yo
<matsubara> herb: 
<matsubara> ok, herb can join in later. let's move on
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs 
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (stub)
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>     * sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
<matsubara>     * ursinha to file a bug about "appserver isn't recovering like it should causing too many oopses"
<matsubara>       * filed bug 360846
<matsubara>     * intellectronica to talk to gmb about bug 269538
<matsubara> ok, Ursinha done hers
<sinzui> wow I do suck
<matsubara> [action] sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
<MootBot> ACTION received:  sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
<sinzui> matsubara: I will do this email IMMEDIATELY after this meeting.
<matsubara> thanks sinzui 
* sinzui starts the subect line now
<intellectronica> matsubara: sorry, i didn't. will try to do that later today
<matsubara> intellectronica: cool. thanks. shall I keep the action item for the next meeting?
<rockstar> me
<intellectronica> matsubara: yes, please
<matsubara> [action] intellectronica to talk to gmb about bug 269538
<MootBot> ACTION received:  intellectronica to talk to gmb about bug 269538
<matsubara> ok, let's move on. thanks sinzui and intellectronica 
<matsubara> hi rockstar 
<rockstar> hi
<matsubara> [TOPIC] * Oops report & Critical Bugs 
<MootBot> New Topic:  * Oops report & Critical Bugs
<matsubara> Ursinha: take the stage please
<Ursinha> I have only one bug for foundations, bug 357307
* Ursinha kicks the bot
<Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/357307
<Ursinha> flacoste, ^
<flacoste> yep, looking at this now
<flacoste> Ursinha: what's the priority on this?
<matsubara> I have one for sinzui: bug 358332
<flacoste> sinzui: do you think salgado could look into that token bug?
<matsubara> curtis, do you agree with the importance I set for https://bugs.edge.launchpad.net/launchpad-registry/+bug/358332 ?
<Ursinha> flacoste, we're had 25 yesterday on lpnet
<Ursinha> about this average a day, oopses count
<sinzui> Well I do not agree with Medium, I do not ever use that status
<matsubara> sinzui: can you reset the importance to a more meaningful value then? 
<sinzui> flacoste: Would you like to take salgado to your team then ;)
<Ursinha> haha
<flacoste> sinzui: you know my opinion on that one :-)
<sinzui> matsubara: Since I have committed to working on series issues, I will try to land a fix for bug 358332 this release
<flacoste> sinzui: and I'll take that as a yes
<matsubara> thank you sinzui 
<sinzui> Yes, I will ask salgado to look at the bug 357307
<matsubara> [action] sinzui to take bug 358332
<MootBot> ACTION received:  sinzui to take bug 358332
<matsubara> [action] sinzui to ask salgado to fix bug 357307
<MootBot> ACTION received:  sinzui to ask salgado to fix bug 357307
<matsubara> All critical bugs are fix committed
<matsubara> so, I think that's all. anything else Ursinha ?
<Ursinha> no matsubara
<matsubara> ok, thanks everyone
<Ursinha> thanks all
<matsubara> I'm going to skip herb's section since he's not here yet.
<mthaddon> matsubara: I'm here
<matsubara> oh
<matsubara> hi mthaddon 
<mthaddon> matsubara: herb's off
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<mthaddon>  - Preparations for new rollout procedure - need to confirm a fair amount of stuff with stub before the next rollout
<mthaddon>  - PostgreSQL upgrade yesterday meant some downtime
<mthaddon>  - Need to decide on whether we're switching the master to wildcherry and if so when
<mthaddon> I think that's it unless there are any questions
<intellectronica> mthaddon: i have a question
<matsubara> what's this new rollout procedure?
<mthaddon> matsubara: SSO related
<mthaddon> intellectronica: sure
<intellectronica> mthaddon: yesterday morning (UTC) forster needed a restart because it was overwhelmed. Spads helped me with that. do you know what happened?
<mthaddon> matsubara: involves splitting out a slave from as standalone so we can continue to serve SSO
<mthaddon> intellectronica: I don't - we've had some issues with forster over the past few weeks
<matsubara> mthaddon: cool.
<mthaddon> intellectronica: i.e. this isn't the first time it's needed a kicking - I'll see if I can look into it a little more
<intellectronica> mthaddon: ok, as long as you're not surprised. sorry i forgot to mention this yesterday
<matsubara> anything else for mthaddon?
<mthaddon> intellectronica: we're moving the incoming email parsing script off there, so hopefully that'll reduce load a little, although I'd be surprised if it was related to that
<flacoste> mthaddon: any idea on the surge in buildbot failures yesterday?
<mthaddon> flacoste: I was off yesterday so I wasn't aware of that, no
<flacoste> mthaddon: it looks it was restarted, i assume by spm since gary nor i did it
<intellectronica> mthaddon: if it's not the cause, then at least it won't stop working when whatever is the cause starts going wild :)
<flacoste> mthaddon: ok, i'll check with spm later
<mthaddon> intellectronica: absolutely :)
<matsubara> flacoste: maybe it was related to the unexpected downtime we had yesterday
<matsubara> flacoste: all over the DC
<flacoste> matsubara: none of this lives in the DC yet
<matsubara> well, but the code buildbot is pulling does :-)
<matsubara> anyway, spm can clarify that later.
<matsubara> thanks everyone
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> I have been on public holidays or ill for the last week, but things still seem to be running which is nice.
<stub> We got a replication lag spike the other night - all the systems coped well while it was cleared. Possibly the rosetta imports being busier than usual. Possibly the rosetta imports where just delayed by something else. I don't think anyone has traced what processes where running at that time yet from the script activity records or the output on launchpad-errors.
<stub> We are going to have to be more agressive at blocking scripts when the system is under load though. I might be able to retrofit this to all our existing scripts by blocking after a commit or rollback if the system is loaded using similar rules to the garbage collector (no transactions open longer than 30 mins and replication lag < 30 seconds).
<stub> Yes, we should switch the master to wildcherry before the next rollout or during the next rollout if we don't mind a few minutes downtime on the SSO server.
<stub> I need to go over rollout procedures with the losas once I know how much of read-only-launchpad lands this cycle.
<stub> EOM
<mthaddon> stub: when do you expect to know how much of read-only-launchpad lands this cycle?
<stub> I don't know yet ;) What would be a good deadline?
<mthaddon> stub: I think we'd need to discuss things at the latest by the end of week 3, so sometime before then?
<mthaddon> in terms of rollout stuff, I mean
<stub> Sure. I was thinking early next week to fix the plan so that works.
<matsubara> anything else for stub?
<matsubara> or anything else before I close?
<matsubara> thanks stub 
<mthaddon> stub: cool, thx
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:29.