## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 09:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
* salgado is now known as salgado-lunch
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
* Notify: salgado is offline (simmons.freenode.net).
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<herb> me
<jtv> me
<sinzui> me
<Ursinha> me
<rockstar> me
<jtv> I'm standing in for henninge today, since he's on sprint.
<matsubara> thanks jtv 
<danilos> me
<BjornT> me
<matsubara> so, if any of you can't make the meeting next week, please coordinate with another teammate to cover for you and add a notice in the Apologies section in the MeetingAgenda page.
<matsubara> flacoste: ping
<intellectronica> me
<matsubara> bigjools: ping
<bigjools> me
<flacoste> me
* henninge (n=henning@port-213-160-23-156.static.qsc.de) has joined #launchpad-meeting
<matsubara> ok, let's move on, stub can join later
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs 
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (DBA contact)
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one - there was, bug 280925
<matsubara>  * intellectronica to work on bug 279561
<matsubara>  * rockstar to check OOPS-1125CEMAIL1
<matsubara>  * bac to take a look at OOPS-1125A165 - bac filed bug 322792
<matsubara>  * Ursinha to check with kiko if any other rollouts will happen this week
<ubottu> Launchpad bug 280925 in launchpad-registry "Project overview page shows obsolete series" [Low,Fix released] https://launchpad.net/bugs/280925
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
<ubottu> Error: Could not parse data returned by Launchpad: The read operation timed out (https://launchpad.net/bugs/279561/+text)
<ubottu> Launchpad bug 322792 in launchpad-bazaar "Attempting traversal past an unknown object causes an OOPS" [Undecided,Fix released] https://launchpad.net/bugs/322792
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
<Ursinha> holy crap
* stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
<Ursinha> that bug is timing out
<Ursinha> or what?
<Ursinha> anyway
<matsubara> intellectronica: any news about the api bug? 
<Ursinha> my items were done
<matsubara> rockstar: what's up about that oops?
<intellectronica> matsubara: sorry, no news yet
<Ursinha> matsubara, he landed a fix for that
<matsubara> Ursinha: who's he?
<matsubara> :-)
<Ursinha> matsubara, rockstar :)
<Ursinha> sorry
<rockstar> matsubara, I got an RC in for that one.
<matsubara> ok, I remember that one. 
<matsubara> so I guess the only pending one is the api bug which is a mistery to everyone
<matsubara> the good news is that intellectronica found out another thing about that bug that might lead to its root cause
<matsubara> intellectronica: thanks for keeping us posted in the report
<matsubara> let's move on
<matsubara> [TOPIC] * Oops report & Critical Bugs 
<MootBot> New Topic:  * Oops report & Critical Bugs
<matsubara> so in today's oops section I'd like to talk about the timeout bugs you guys are working for the LPW
<matsubara> https://dev.launchpad.net/PerformanceWeeks/February2009
<matsubara> I'm going to review all the landings related to LPW work and add to that page.
<matsubara> so if you wanna help, point me to revision numbers on RF related to that work
<jtv> matsubara: bug 324264 is now Fix Committed.
<ubottu> Launchpad bug 324264 in rosetta "Speed up +translations" [High,Fix committed] https://launchpad.net/bugs/324264
<sinzui> matsubara: r7705 for Bug: 325321
<Ursinha> jtv, great
<matsubara> intellectronica, BjornT : any news about the bug number 1 time out?
<sinzui> matsubara: EdwinGrubbs will land his branch today
<danilos> work on bug 302798 is in progress in different ways (there's a commit from 323something which disabled external suggestions to give us a better idea on how stuff is working, and henning is working on removal of obsolete translations which will reduce our DB size by ~33%)
<ubottu> Launchpad bug 302798 in rosetta "Timeout on +translate page" [High,Triaged] https://launchpad.net/bugs/302798
<BjornT> matsubara: me, intellectronica, and allenap are working on it
<jtv> danilos: hey, I was putting that paragraph together!
<BjornT> matsubara: allenap is looking at reducing the time it take to render the comments, possibly by not showing all by default
<danilos> jtv: ok, I'll let you handle it all from now on :)
<BjornT> matsubara: intellectronica is working on loading the subscribers portlet in a different request
<danilos> matsubara: you can have full trust in jtv as far as PW is concerned :)
<jtv> *cough*
<BjornT> matsubara: and i'm working on optimizing code, based off profiling information
<matsubara> danilos: I do! he's been very helpful with the status updates
<BjornT> matsubara: also, intellectronica's inital tests on dogfood were successful, reducing the time quite a lot :)
<matsubara> great
<flacoste> matsubara: i'm working on bug 316881 (which isn't an OOPS per-se, but related to performance anyhow)
<ubottu> Launchpad bug 316881 in launchpad "Page headers not suitable for HTTP caching" [High,In progress] https://launchpad.net/bugs/316881
<matsubara> stub: there's an email from jono asking for some help with the +project-cloud oops, so if you could help out there, would be awesome
<matsubara> thanks flacoste, i'll add it to the page
<stub> I've replied on the bug. Not sure if I'm helpful though.
<matsubara> stub: cool. thank you
<matsubara> bigjools: news in soyuz. how about the one muharem is taking care of?
<matsubara> s/./?/
<bigjools> matsubara: it's not going so well unfortunately
<bigjools> I don't expect any progress this week
<matsubara> [action] matsubara to add 316881 to foundations section in LPW wiki page
<MootBot> ACTION received:  matsubara to add 316881 to foundations section in LPW wiki page
<matsubara> bigjools: why is that? 
<bigjools> matsubara: the first attempt to fix it failed.  he's also been at the distro sprint this week
<matsubara> bigjools: oh right. well, you guys are excused since the whole team is sprinting and you already landed 2(?) timeout fixes :-)
<bigjools> matsubara: thanks :)
<matsubara> I guess that's it from me. Ursinha, anything else?
<Ursinha> matsubara, no, the pending oops for soyuz I already talked with bigjools
<bigjools> yeah, get edge updating again and we'll see how it went
<matsubara> great. thanks everyone!
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> - 2009-01-30 - Friday we updated lpnet, edge and the scripts servers to to r7676.
<herb> - 2009-02-04 - Yesterday we updated codebrowse to r43
<herb> - I feel like I'm starting to sound like a broken record... But we're still being bothered daily, often multiple times, by bug #156453 and bug #118625 which seem to be related.
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<herb> - We also continue to run into issues associated with bug #260171
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
<flacoste> herb: i herd mwhudson was working on loggerhead performance this week
<flacoste> as part of LPW
<matsubara> yes!
<sinzui> herb: There is a sprint in the planning to fix that too
<Ursinha> yes, he is
<matsubara> mwhudson is indeed working on that
<flacoste> herb: and well-placed sources also told me that a sprint is being organized to fix those damn issues
<flacoste> herb: so there is hope!
<herb> excellent. a fix would be huge for the LOSAs
<rockstar> herb, mwhudson is on the verge of insanity tracking loggerhead issues down.
<herb> thanks for all the work on that. it is much appreciated.
<matsubara> thanks herb 
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> The production dbs seem to be ticking away nicely.
<stub> Staging db updates are being disabled by the losas for some testing, so expect a drop in timeout OOPSES.
<stub> I've had some db patches for this cycle come through already, which is great.
<stub> 3
<stub> 2
<stub> 1
<matsubara> all right. thanks stub 
<flacoste> what about the staging issues?
<matsubara> herb: ^
<flacoste> staging isn't available at the moment
<herb> it's restoring
<flacoste> but we didn't understand why it failed?
<flacoste> or did we?
<matsubara> herb: how long will it take to restore?
<flacoste> and it's just that gmb is done with the testing and we can resume normal staging updates?
<herb> matsubara: should be back up within the next couple of hours.
<Ursinha> herb, about 10 hours ago I was talking with spm and he was unsuccessful to put staging on again
<herb> flacoste: upgrade.py failed because there will still connections open to the staging db.  This left the DB in an indeterminate state, so we had to go through the full restore process with a new copy of the staging DB.
<gmb> flacoste: I'm not done yet.
<gmb> flacoste: I need staging to be up for that
<flacoste> so this means that we are still screwed?
<stub> This is all useful information for read only launchpad btw.
<flacoste> we either need to fix upgrade.py to work without a db restore
<flacoste> or to turn off staging ugprades for the duration of gmb's test
<herb> flacoste: upgrade.py works fine
<herb> and we'll need to turn off upgrades while gmb's testing, per stubs note above.
<flacoste> why were there still connections open?
<stub> upgrade.py cannot work when there are active db connections, as upgrade requires exclusive locks on all the replicated db tables.
<herb> flacoste: the rollout process shuts down the app servers, but there are still potentially cron scripts running, etc.
<flacoste> why usually is this not a problem then?
<stub> Because the restore, replicate and upgrade process is done as a fresh db. Once it is finished, it is swapped into place.
<flacoste> ah right!
* bigjools has to run, will catch scrollback later if you need anything from me
<flacoste> so when I said fix upgrade.py, i should have said 'fix rollout process'
<flacoste> so I guess it's best to simply disbale upgrade for now
<flacoste> i also heard that gmb might do his tests elsewhere
<flacoste> so that might becomes a moot issue
<flacoste> but like stub said, very instructive for read-only launchpad
<gmb> flacoste: Well, that's an embryonic idea right now. I still need a staging / demo machine for the forseeable future.
<matsubara> I want to make an action item for the fix rollout process thing
<matsubara> not sure who would be responsible for that
<matsubara> losas?
<herb> whoa
<flacoste> with the help of stub probably
<matsubara> [action] losas and stub to fix rollout process to avoid the staging restore problems
<MootBot> ACTION received:  losas and stub to fix rollout process to avoid the staging restore problems
<herb> there isn't anything inherently wrong with the rollout process.
<herb> but ok
<matsubara> thanks everyone
<matsubara> anythign else before I close?
<flacoste> well, it's currently not reliable if we don't do a DB restore it seems
<flacoste> nope
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
<matsubara> #endmeeting
<MootBot> Meeting finished at 09:36.