## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 09:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<henninge> me
<Ursinha> me
<matsubara> Ursinha, flacoste, bigjools, intellectronica, herb
<bigjools> me
<herb> me
<matsubara> bac, ping
<flacoste> me
<Ursinha> matsubara, already answered
<intellectronica> me
<matsubara> rockstar, hi
<rockstar> me
<rockstar> matsubara, hi
<bac> me
<matsubara> ok, stub can join later. everyone else is here.
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs 
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (DBA contact)
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * stub to investigate the fix to avoid staging restore problems
<matsubara>  * matsubara to chase rockstar about a fix for OOPS-1138CEMAIL12
<matsubara>     * asked jml about this. It's bug 326056 and had importance raised.
<matsubara>  * cprov and bigjools to investigate OOPS-1145EA14
<matsubara>  * Ursinha to file bugs:
<matsubara>     * Bug 333072: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189
<matsubara>     * Bug 333071: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
<ubottu> Launchpad bug 326056 in launchpad-bazaar "OOPS on BadStateTransition when reviewing code by mail" [High,Triaged] https://launchpad.net/bugs/326056
<ubottu> Launchpad bug 333072 in soyuz "AttributeError OOPS on Build:+index" [Undecided,Invalid] https://launchpad.net/bugs/333072
<ubottu> Launchpad bug 333071 in soyuz "AssertionError OOPS on +copy-packages" [High,Triaged] https://launchpad.net/bugs/333071
<bigjools> 333072 is invalid
<matsubara> bigjools, any news about 333071?
<bigjools> yes, it's not too serious, we've set it for 2.2.3
<bigjools> it's a corner case in the copying
<bigjools> despite the doom-mongering error message
<matsubara> ok. thanks bigjools 
<matsubara> [action] matsubara to chase stub about staging restore problems
<MootBot> ACTION received:  matsubara to chase stub about staging restore problems
<matsubara> [TOPIC] * Oops report & Critical Bugs 
<MootBot> New Topic:  * Oops report & Critical Bugs
* matsubara hands Ursinha the mic
* Ursinha looks
* rockstar runs
<Ursinha> registry, foundations, code and bugs: oopses for you
<Ursinha> Registry:-
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153E919
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153A1135 (or foundations, not sure)
<Ursinha> Foundations:-
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153D667
<Ursinha> Code:
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153E919
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153A1135
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1152XMLP1
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153D667
<Ursinha> Bugs:
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1152EA162
<Ursinha> ~
<Ursinha> rockstar, ha!
<Ursinha> rockstar, have you seen this one: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1152XMLP1?
<rockstar> Ursinha, looking at all of them now.
<Ursinha> rockstar, you can just look at code's one :)
<Ursinha> sinzui, hi
<Ursinha> sinzui, I'm not sure if https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153A1135 is foundations or registry
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153A1135
<matsubara> Ursinha, looks like registry
<intellectronica> Ursinha: strange. do you see lots of those?
<bac> Ursinha: yes, looks like registry
<Ursinha> intellectronica, no, actually not
<Ursinha> intellectronica, but never saw one of those before
<Ursinha> so better bring to attention
<matsubara> intellectronica, Ursinha that one looks like caused by the rollout
<sinzui> Ursinha I don't know the answer either. I will look into it and assign it. I suspect salgado-afk is working on
<Ursinha> matsubara, even in the time it happened?
<intellectronica> matsubara: i also thought so
* salgado-afk is now known as salgado
<intellectronica> but it is quite early
<Ursinha> intellectronica, I've discarded the rollout possibility because of its timestamp
<Ursinha> sinzui, thanks for that
<matsubara> yeah, too early to be caused by the rollout. 
<Ursinha> intellectronica, can you take a look then, please?
<rockstar> Ursinha, I'll have to investigate our oops.  It's the XML-RPC server, and it requires the sacrifice of a virgin goat.
<matsubara> check OSAs incident log to see if something happened during that time
<intellectronica> so, this isn't really a bugs oops, but i don't know whether it's rollout-related or not. fwiw it's more than three hours before rollout, so it's hard to see how it would be related
<Ursinha> rockstar, oh, I have a bunch here in my backyard if you need some
<rockstar> Ursinha, :)
<Ursinha> intellectronica, I'll do what matsubara suggested
<matsubara> [action] ursinha to check OSAs incident log to help identify cause of OOPS-1152EA162
<MootBot> ACTION received:  ursinha to check OSAs incident log to help identify cause of OOPS-1152EA162
<Ursinha> thanks intellectronica and matsubara
<matsubara> [action] rockstar to investigate xmlrpc oops OOPS-1152XMLP1
<MootBot> ACTION received:  rockstar to investigate xmlrpc oops OOPS-1152XMLP1
<Ursinha> flacoste, hi
<henninge> Translations is happy, that POFile:+translate dropped from the timeout top ten now ..
<henninge> btw
<henninge> ;)
<Ursinha> henninge, indeed, congrats to translate team :)
* stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
<Ursinha> translations
<Ursinha> there he is :)
<henninge> Ursinha: thank you, I will pass it on.
<Ursinha> sinzui, about the other oops
<stub> Sorry - on a call and didn't realize the time
<sinzui> bac: can you look at it.
<bac> Ursinha: they seem to be related (acting for sinzui today)
* sinzui is in another meeting
<flacoste> hmm
<flacoste> i'd say registry
* cumulus007 (n=sander@unaffiliated/cumulus007) has joined #launchpad-meeting
<bac> yes, i think registry for both
<flacoste> Ursinha are you talking about OOPS-1153A1135?
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153A1135
<Ursinha> bac, hi :) so, can you take a look in both oopses? do you need me to file bugs about them?
* Ursinha looks
<bac> Ursinha: yes i'll look at them both
<bac> i can open the bugs
<bac> unless you need the karma
<Ursinha> flacoste, no, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153D667
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153D667
<Ursinha> bac, haha, no
<flacoste> Ursinha: that's also a registry query
<Ursinha> [action] bac to file bugs and take care of https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153E919 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1153A1135
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153E919
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153A1135
<matsubara> [action] bac to file bugs for OOPS-1153E919 and OOPS-1153A1135 
<MootBot> ACTION received:  bac to file bugs for OOPS-1153E919 and OOPS-1153A1135
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153E919
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153A1135
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153E919
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153A1135
<bac> wow, y'all are insistent today!  :)
<Ursinha> :)
<Ursinha> flacoste, hm.
<Ursinha> thanks
<Ursinha> bac, can you take a look at that too?
<bac> which?
<Ursinha> promise not to paste the oops again
* danilo-afk is now known as danilos
<Ursinha> bac, https://devpad.canonical.com/~jamesh/oops.cgi/1153D667
<Ursinha> I tried :)
* bac looks
<bac> yes
<Ursinha> bac, thanks
<Ursinha> that's all from me from the oops land
<matsubara> [action] bac to also file a bug and take care of OOPS-1153D667
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153D667
<MootBot> ACTION received:  bac to also file a bug and take care of OOPS-1153D667
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1153D667
<matsubara> ok, thanks everyone.
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<Ursinha> there's one critical bug, though
<Ursinha> argh
<Ursinha> bad bad timing
<herb> shall I wait for the critical bug?
<matsubara> danilo is handling the critical bug, so won't duplicate what's in the bug report.
<Ursinha> herb, just a second, let me check with henninge
<matsubara> it's bug 334787
<Ursinha> matsubara, okay, if you say so
<ubottu> Launchpad bug 334787 in rosetta "Ubuntu packagers are not translation editors (assertion error)" [Critical,In progress] https://launchpad.net/bugs/334787
<matsubara> let's move on
<Ursinha> go ahead herb, thanks
<herb> 2009-02-20 - We had an issue that may have caused some users to experience intermittent outages on Launchpad. I worked with joey and flacosted to find the issue. joey's notes were sent to the list. I would be interested in hearing any updates we might have on this issue.
<herb> 2009-02-21 and 2009-02-22 - It appears we had bit of buggy code land on edge that caused a performance problem on both edge and production. The revision was backed out and I believe the code has been fixed.
<herb> 2009-02-26 - We rolled out 2.2.2 based on r7763
<herb> We continue to see problems relating to bug #156453 and bug #118625. So much so that we're going to start bouncing codebrowse regularly to hopefully head off any issues. I want to emphasize that this will be masking the problem and we really do need to find the root cause and fix it.
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<herb> Bug #260171 continues to creep up regularly (every few days). This is already morked as high and I know that mwhudson's plate is full with codebrowse issues, but can we get an update on this one?
<ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
* herb somehow managed to change flacoste into a verb.
<danilos> matsubara, Ursinha: I am running tests on the critical bug fix, will let you know once it has landed
<flacoste> i saw!
<bac> i've been flacosted!
<matsubara> danilos, thanks
<Ursinha> thanks danilos
<matsubara> rockstar, can you bring up the codebrowse issue to the code team?
<rockstar> matsubara, everyday.  :)
<matsubara> rockstar, thanks :-)
<rockstar> Codebrowse is being ACTIVELY worked on.  It'd be nice if we knew what the issues is.  Right now, we're just fixing things and hoping that was the problem.
<herb> rockstar: let the losas know if there is anything we can do to help.
<rockstar> herb, we certainly will.
<stub> Should we be bringing in any outside help to intrument, test and diagnose the issue?
<matsubara> herb, anything happened to the DB during the time of this OOPS-1152EA162?
<matsubara> or maybe stub might know ^
<herb> matsubara: nothing in the incident log.
<stub> matsubara: That is one of the connection reaper scripts kicking in
<herb> matsubara: I think that's also on the void between LOSAs.
<herb> ah, there we go.
<stub> We kill connections idle in a transaction more than a few hours (and should be more agressive), and appserver connections that have been in a transaction for more than 2 minutes.
<Ursinha> stub, I see
<matsubara> stub, ok. so if we start seeing too many of those, we have a problem somewhere and a few is kinda normal?
<stub> The notification gets sent to the error-reports list (where we can confirm that this is indeed what happened)
<matsubara> stub, aha. that's better. I'll chase the lp-errors for that one
<matsubara> s/lp-errors/lp-errors list/
<stub> If we see many of them, we have a problem. One is probably a problem - appserver requests taking two minutes on the db means we need to investigate why the normal timeout mechanisms didn't work.
<matsubara> [action] matsubara to look lp-errors list to determine cause of OOPS-1152EA162
<MootBot> ACTION received:  matsubara to look lp-errors list to determine cause of OOPS-1152EA162
<matsubara> right. thanks for the explanation
<stub> -1 second non-sql time, 0 seconds total time indicates a problem at the appserver? The request never got started?
<matsubara> I'll file a bug about that one and we can discuss there
<stub> hmm... might be a reconnection bug - perhaps the previous request handled by that thread got killed?
<stub> I don't know if we Retry on DisconnectionError exceptions, or if it is a good idea in all cases.
<matsubara> ok
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> and thanks herb and stub 
<stub> New hardware exists and is being brought online by IS. I've realized I might need to tweak the db maintenance scripts (upgrade.py, security.py etc.) to cope with a third replica - I think it only copes with a single master and slave at the moment.
<stub> Staging can be moved by the LOSAs as soon as the hardware is available and they have time, which will move that load from the production systems.
<stub> I assume the rollout went fine as far as the db upgrade procedure goes.
<herb> I assume it did too. I didn't hear any complaints from my colleagues.
<matsubara> stub, great news! with the new hardware we won't have the staging restore problems anymore?
<herb> stub: what's the plan with the 3rd replica?
<stub> The staging restore problems should no longer be a problem.
* herb feels like he missed something
<stub> herb: We can start by pointing half the appservers at the new slave when it is online. We really should get a connection pool/load balancer thingy though running like pgbouncer, pgpool 1 or 2.
<herb> stub: gotcha
<stub> herb: I realized just now though that upgrade.py won't apply patches to a third replica, which would be bad. So that needs to be fixed.
<herb> yeah. that's important.
<stub> Or actually, slonik may take care of all that. I need to confirm anyway.
<stub> I forget and it is too late for my brain :)
<stub> erm... late as in evening
<matsubara> all right. I guess that's all unless there are questions for stub
<matsubara> thanks stub 
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 09:42.
<intellectronica> thanks matsubara
<flacoste> hey
<flacoste> matsubara: question
<flacoste> do we need a new roll-out?
<flacoste> and i think it applies to everyone here
<matsubara> flacoste, I was on vacation and need t ocheck that
<flacoste> anyone requires a new roll-out?
<matsubara> but I think there's at least danilos' bug to re roll
<bac> flacoste: i don't know of any issues for us
<danilos> matsubara, flacoste: yes
<stub> I thought it was policy to let enough bugs through qa to require a rerollout?
<flacoste> we're getting better at QA stub
<flacoste> even the code team weren't that late this cycle :-)
<matsubara> ok, so we'll need a re-roll for translations. need to check for the other teams, but so far, there's nothing on the radar
<stub> We need a counter somewhere - 'Launchpad has been running for n days without need to a release critical patch'
<Ursinha> stub, :)
<matsubara> I think that's all then. thanks everyone