## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<Ursinha> roll call,roll call
<matsubara> my firefox died
<sinzui> me
* stub belches
<matsubara> hang on a second please
<henninge> me
<Ursinha> poor matsubara
* jml eavesdrops
* bigjools wafts stub's belch away
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<rockstar> ni
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<Ursinha> meeee
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<bigjools> me
<stub> me
<henninge> me again
<intellectronica> i
<matsubara> flacoste, hi
<flacoste> me
<matsubara> herb, hi
<herb> me
<matsubara> ok, everyone here.
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (stub)
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * matsubara to chase rockstar about failure on updatebranches script
<matsubara>  * stub to give a try on bug 354593 with mars help if needed
<matsubara>  * stub to fix bug 310818
<matsubara>  * mars to take a look at OOPS-1307J16
<matsubara>  * Discuss the solution proposed by gary_poster after the meeting, about ExpatErrors and bug 403606
<matsubara>  * mars and stub to discuss the Disconnection and OperationalErrors after the meeting
<jml> me
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<ubottu> Launchpad bug 310818 in launchpad-foundations "Oops report does not always log timed-out query" [High,In progress] https://launchpad.net/bugs/310818
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<Ursinha> yay, jml
<matsubara> I suck, I didn't chase rockstar about the updatebranches script failures
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<rockstar> matsubara, I thought we agreed that mwhudson would be better to chase on it.
<jml> matsubara, got a URL for the failure?
<matsubara> otoh, the script is not failing anymore...
<rockstar> matsubara, I know mwhudson was looking at it on his Tuesday.
<rockstar> jml would be good to ask as well.
* cprov is now known as cprov-lunch
<matsubara> rockstar, all right. I'll talk to jml and mwhudson later on today
<matsubara> [action] * matsubara to chase mwhudson/jml about failure on updatebranches script
<MootBot> ACTION received:  * matsubara to chase mwhudson/jml about failure on updatebranches script
<rockstar> matsubara, jml is here right now. :)
<matsubara> jml, I'll get you an url for the scripts after the meeting. I need to trawl my emails to find it
<jml> matsubara, ok. thanks.
<matsubara> stub, how's 354593 fix coming along?
<flacoste> why is this High again?
<matsubara> I wonder if mars had time to look over OOPS-1307J16
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<matsubara> flacoste, do you know ^?
<flacoste> hmm, i put it as such
<flacoste> any reason it should be?
<matsubara> flacoste, according to the bug history you made it high :-)
<flacoste> debranding of the SSO is a U1/ISD affair anyway
<stub> matsubara: Slow. I need to discuss with people how to actually do it - maybe next week on the sprint if I get time.
* sinzui agrees with flacoste
<flacoste> stub: i think we should try to get stu and James to do it :-)
<flacoste> especially, stu, it would be a test good case for transfer knowledge
<stub> Anything that means I don't have to work out how ZPT macros works is fine by me.
<flacoste> +1
<matsubara> Ursinha, what's up with  "Discuss the solution proposed by gary_poster after the meeting, about ExpatErrors and bug 403606"?
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<Ursinha> matsubara, the ExpatErrors were being discussed by mars and gary
<gary_poster> matsubara: that;s now registry.  it actualy is a legitimate oops
<matsubara> [action] stub to delegate bug 354593 to ISD
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<MootBot> ACTION received:  stub to delegate bug 354593 to ISD
<gary_poster> it indicates a problem with mailman integration
<sinzui> I will ask barry to look into bug 403606
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<matsubara> stub, You recently fixed a DisconnectionError bug. was it related to the errors you discussed with mars? that action item is now done?
<matsubara> thanks sinzui and gary_poster 
<gary_poster> :-)
<stub> matsubara: I landed code to log OOPS reports on DisconnectionError before retrying the request. Is that what you mean?
<matsubara> stub, I mean: "* mars and stub to discuss the Disconnection and OperationalErrors after the meeting"
<Ursinha> stub, is that what caused the TransactionRollbackError oopses?
<stub> We discussed. I don't recall much about the conversation though :)
<matsubara> :-)
<stub> Ursinha: That fix was, yes. I've got another branch that turns the volume down so we don't log the TransactionCommitError's
<matsubara> [action] sinzui to ask barry to fix bug 403606
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
<MootBot> ACTION received:  sinzui to ask barry to fix bug 403606
<Ursinha> stub, good, I filed bug 409907 for that
<ubottu> Launchpad bug 409907 in launchpad-foundations "TransactionRollbackErrors may prevent us to detect real issues" [Undecided,New] https://launchpad.net/bugs/409907
<matsubara> Ursinha, is there a bug for OOPS-1307J16?
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<Ursinha> matsubara, not that I opened one, because we needed to know what was going on over there
<Ursinha> to open the bug
<Ursinha> so mars was going to investigate that
<Ursinha> I don't recall having those anymore
<matsubara> [action] ursinha to chase mars about OOPS-1307J16 and file a bug about it
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<MootBot> ACTION received:  ursinha to chase mars about OOPS-1307J16 and file a bug about it
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
<matsubara> I think that's all for last meeting's action items
<matsubara> thanks everyone
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> there are two issues to discuss
<Ursinha> one was about bug 409907, that I already mentioned to stub and it's being handled
<ubottu> Launchpad bug 409907 in launchpad-foundations "TransactionRollbackErrors may prevent us to detect real issues" [Undecided,New] https://launchpad.net/bugs/409907
<Ursinha> the other is about the select replication_lag() timeouts we're having
<Ursinha> mthaddon also reported problems that we don't know if are related to that
<Ursinha> I don't know if there's much to be discussed at this point, because it seems we need to fix oops reports first to be able to see the real problem here
<Ursinha> is that correct stub:
<Ursinha> ?
* noodles775 has quit (Read error: 54 (Connection reset by peer))
<matsubara> should we request a CP for the branch that fixes the oops log?
<intellectronica> given that we're skipping a release, that's probably a good idea
<Ursinha> flacoste, I've spoken with jtv yesterday about those,and he also said that was unlikely to be his changes fault (possible but unlikely)
<stub> I landed code today that should tell us more about if the timeout is actually occuring due to blocking on the database, or elsewhere.
<Ursinha> s/his/translations/
<Ursinha> stub, should we request a CP?
<flacoste> yeah, i really think a CP is a good idea
<Ursinha> (please please)
<matsubara> [action] stub to request CP for his branch that fixes oops logging
<MootBot> ACTION received:  stub to request CP for his branch that fixes oops logging
<Ursinha> cool
<Ursinha> we have two critical bugs, already fix committed
<Ursinha> so, good
<matsubara> cool
<Ursinha> about the failing scripts
<matsubara> we had some scripts failing this week
<matsubara> nightly, productreleasefinder and garbo-hourly
<matsubara> and rosetta-poimport too
<matsubara> nightly was already addressed by jtv
<Ursinha> matsubara, productreleasefinder isn't expected to fail anymore? sinzui?
<matsubara> as a rosetta script was taking too much time and jtv will remove it from nightly and add a cronjob for it
<sinzui> Ursinha: no, but the errors is see are not failures...the script was not run
<matsubara> stub, do you know why garbo-hourly is failing?
<stub> Its failing?
<sinzui> matsubara: many scripts are not running because of one log process
<matsubara> henninge, rosetta-poimport failed on the 5th. can you investigate and reply to the list?
<sinzui> s/log/long/
<Ursinha> matsubara, it's not being run, it seems
* noodles775 (n=miken@canonical/launchpad/noodles775) has joined #launchpad-meeting
<henninge> matsubara: sure, I will.
<matsubara> stub, I got a few emails: "Scripts failed to run: loganberry:garbo-hourly" 
<sinzui> Ursinha: matsubara there is some traffic about this. spm reported the long running prcess a weeks ago. I has asked why the prf had not run
<matsubara> and no replies to the list, so I'm asking here
<matsubara> thanks henninge 
<Ursinha> matsubara, actually stub repklied
<Ursinha> *replied
<stub> Oh - there were some blocked runs because the rosetta export-to-branch script was running in a 5 hour long transaction
<stub> So the script blocks because it doesn't want to make anything worse.
<matsubara> [action] henninge to investigate rosetta-poimport script failure on the Aug 5th and report back to the list
<MootBot> ACTION received:  henninge to investigate rosetta-poimport script failure on the Aug 5th and report back to the list
* salgado is now known as salgado-lunch
<Ursinha> so I guess it's ok
<Ursinha> that's all for this section
<Ursinha> from me
<Ursinha> thanks everyone
<Ursinha> !
<matsubara> all right. thanks everyone
<Ursinha> you can move on matsubara
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> 2009-07-31 - Rolled out r8323 to bzrsyncd
<herb> 2009-08-05 - Cherry picks for code imports, lpnet* and the script server.
<herb> Our monitoring system has been timing out in connecting to the app servers more often this week. Admittedly its timeout is set lower than the OOPS timeout. But we've also been noticing higher load on the app servers as well. This was discussed by Ursinha during the oops/critical bugs/broken scripts section.
<herb> There's currently 1 cherry pick and 1 database query awaiting (dis)approval.
<herb> The LOSAs currently have 14 bugs marked high and triaged. Only 1 of which is assigned to someone and targeted for a release. We would be grateful if we saw some movement on these.
<herb> We're currently running with a single slave in preparation for the sprint next week.
<mthaddon> also wanted to check that there should be a cherry pick request for the cowboyed storm change to lpnet9 and lpnet10 (per the production status wiki page)
<flacoste> cowboyed storm change?
<mthaddon> flacoste: https://pastebin.canonical.com/20503/ under eggs/storm-0.14salgado_storm_launchpad_288_308-py2.4-linux-i686.egg
<flacoste> mthaddon, herb: i'll look at the LPS to approve/decline
<flacoste> right
<matsubara> herb, do you keep that list of 14 bugs somewhere? in a wiki page or have a tag to group them?
<flacoste> mthaddon: the cherry pick would simply be to update that dependency
<herb> matsubara: bugs.launchpad.net/~canonical-losas
<mthaddon> flacoste: well in any case, the CP that was requested (and performed) yesterday overwrote it, so it needs to be formalised so other CPs don't overwrite it again
<flacoste> sinzui: can salgado makes an appropriate CP request?
<sinzui> Yes
<flacoste> it's simply a new upload to download-cache with a versions.cfg change
<matsubara> sinzui, flacoste, intellectronica, rockstar: Could you take a look at herb's bug list (bugs.launchpad.net/~canonical-losas) and see what your teams can do about the high ones in the short term?
<flacoste> ok
<herb> clearly we're not looking for all of them to be fixed by the next meeting (though that would be great ;)
<herb> just mostly would like to know they're staying on the right radars and are being worked on as appropriate.
<matsubara> cool
<matsubara> anything else for herb?
<intellectronica> herb: so, basically, these are mostly bugs which will make life easier for you when fixed?
<sinzui> bug 348722 should become invalid when we update all pmt teams to become true private teams
<ubottu> Launchpad bug 348722 in launchpad-code "Set default branch visibility to "forbidden" if any team set to 'Private'" [High,Triaged] https://launchpad.net/bugs/348722
<herb> intellectronica: some of them are geniune operational issues, some of them are quality of life issues for the LOSAs
<sinzui> There should be no private-membership teams at the start of week 1
<intellectronica> cool, sure, we'll take a look and see if there's any low hanging fruit
<sinzui> barry will be working with the losas on August 11 to fix bug 325962
<ubottu> Launchpad bug 325962 in launchpad-registry "lp-mailman startup is blocking on a pid file in the wrong directory" [High,Triaged] https://launchpad.net/bugs/325962
<herb> sinzui: that was the one that was assgned and targetted at a release.
<sinzui> herb, many times
<herb> assigned even
<herb> heh
<matsubara> all right. I think that's it
<sinzui> herb it failed my rules that bug is not high if it is not worked on by all parties in 3 months
<herb> thanks
<matsubara> thanks herb and everyone
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> We set off some alerts when the poimport script and PostgreSQL decided that lots of disk space should be used. We see some smaller spikes, which is just PG using disk to store intermediary results, but this time it was large enough to set of the alarms.
<stub> We have seen this once before, and in neither case have we been able to repeat it. My best hypothesis is the planner statistics triggering a really bad query plan, so I'll bump the planner statistic sample size on the production dbs in case this stops future occurances.
<matsubara> henninge, maybe the last rosetta-poimport failure was related to that ^
<henninge> matsubara: I believe we already know what it was about and it may be related to that.
<henninge> matsubara: I'll talk to the guys.
<matsubara> henninge, cool. thanks
<matsubara> stub, anything else?
<stub> Not that I can think of
<matsubara> all right. thank you stub 
<matsubara> I guess that's all for today
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:44.