## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 09:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<henninge> me
<intellectronica> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<herb> me
<Ursinha> me
<stub> me
<matsubara> sinzui, flacoste?
<rockstar> me
<flacoste> me
<matsubara> bigjools: ?
<bigjools> meeeeeeee
<sinzui> m,e
<sinzui> me
<matsubara> all right, everyone is here
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
* sinzui hates windows that steal focus
<matsubara>  * Next meeting
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs 
<matsubara>  * QA pending items
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (DBA contact)
<matsubara> [TOPIC] * Next meeting
<MootBot> New Topic:  * Next meeting
<matsubara> same time next week?
<matsubara> I'll be on vacation, so Ursinha will take over
* bigjools is out for 3 weeks
<Ursinha> o/
<matsubara> [action] matsubara to add himself and bigjools to the apologies list
<MootBot> ACTION received:  matsubara to add himself and bigjools to the apologies list
<matsubara> bigjools: can you arrange for someone in soyuz to cover for you?
<bigjools> matsubara: yep
<matsubara> bigjools: thanks
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * matsubara email stub asking for the DBA report
<matsubara>  * matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324
<matsubara>  * rockstar to talk with code team about their pending QA items
<matsubara>  * henninge  to talk with translation team about their pending QA items
<matsubara>  * Ursinha to talk with bigjools about one account on dogfood machine to do some testing
<matsubara>  * rockstar to handle bug 271879 today (12-04)
<matsubara>  * Ursinha to watch timeouts in the next days to see if they continue to happen this often
<matsubara>  * rockstar to manage having something reported on bug 118625 and bug 156453
<matsubara>  * Ursinha to watch for strange OOPSes on edge in the next days
<ubottu> Error: Launchpad bug 2 could not be found
<ubottu> Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324
<ubottu> Bug 271879 on http://launchpad.net/bugs/271879 is private
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<Ursinha> oh yeah
<Ursinha> I've opened a RT to get access to mawson
<rockstar> matsubara, first item is talked about everyday now.
<matsubara> I did both of mine and gmb added the query to the the LPPS page effectively hiding the problematic bug
<Ursinha> matsubara, great
<Ursinha> I've watched for the oopses and nothing that bizarre came up
<rockstar> matsubara, the fix for the timeout bug is approved, but we need to do some query testing.
<matsubara> thanks rockstar. Is the daily nagging helping QA testing?
<rockstar> matsubara, my team doesn't like nagging, but they're getting to it.
<matsubara> Ursinha: anything unusual about the strange OOPS and timeouts?
<Ursinha> matsubara, nothing that's not reported and being taken care of
<matsubara> henninge: how's the daily nagging about pending QA items going? is it helping your team?
<henninge> matsubara: Well, currently all items have been tested that could have been.
<henninge> matsubara: danilo is at UDS right now and jtv doesn't have anything else to test.
<matsubara> great. thanks henninge 
<henninge> matsubara: we'll keep reminding ourselves ...
<matsubara> cool
<matsubara> moving on
<matsubara> [TOPIC] * Oops report & Critical Bugs 
<MootBot> New Topic:  * Oops report & Critical Bugs
<Ursinha> okay
<matsubara> Ursinha: do you want me to do it?
<matsubara> all right, go ahead
<Ursinha> matsubara, just saw sinzui was talking to salgado about bug 304113
<ubottu> Launchpad bug 304113 in launchpad-registry "Milestone:+index page timing out" [High,Triaged] https://launchpad.net/bugs/304113
<Ursinha> oops, this wasn't for you matsubara, sorry :)
<sinzui> Ursinha: we will start working on it in a few hours
<Ursinha> sinzui, great, thanks
<Ursinha> henninge, can you ask someone on translations to set priority and importance on bug 302798, please?
<ubottu> Launchpad bug 302798 in rosetta "Timeout on +translate page" [Undecided,New] https://launchpad.net/bugs/302798
<henninge> Ursinha: yes
<Ursinha> henninge, thank you :)
<matsubara> priority and importance ?
<Ursinha> erm
<matsubara> we have only importance :-)
<henninge> status and importance
<Ursinha> status and importance, sorry :)
<henninge> Ursinha: I knew what you meant ... ;)
<matsubara> anything else Ursinha ?
<Ursinha> rockstar, you
<rockstar> Me?!
<Ursinha> rockstar, do you know anything about bug 305367
<ubottu> Bug 305367 on http://launchpad.net/bugs/305367 is private
<Ursinha> ?
<Ursinha> grrr
<Ursinha> it's a critical one
<Ursinha> the only one we have
<Ursinha> it's really critical?
<rockstar> Ursinha, I don't think they are critical.
<rockstar> activereviews is for your own code.
<Ursinha> rockstar, stub marked it critical
<rockstar> I'm working on the pages to generate a list of reviews you need to vote on.
<Ursinha> rockstar, right.
<rockstar> I'm going to bring it down from critical, anh talk with stub.
<stub> I marked it critical because I couldn't see a work around and might lose things. Feel free to point me at a work around :)
<Ursinha> rockstar, thank you very much
<rockstar> stub, no work around.
<rockstar> stub, I think your people should be contacting you to tell you about them.
<Ursinha> I'm done after that
<rockstar> That should be the workaround right now.
<stub> rockstar: Not really an option with the db review windows - I'll use a wiki page next cycle.
<rockstar> stub, next cycle you'll have the page you're looking for.  I'm working on it now.
<rockstar> But activereviews is for seeing the code you've proposed for review and its status.
<stub> Ok.
<matsubara> all right, moving on.
<matsubara> thanks everyone
<Ursinha> thanks guys
<matsubara> [TOPIC] * QA pending items
<MootBot> New Topic:  * QA pending items
<Ursinha> right
<Ursinha> so we have a lot of items to be tested
<rockstar> We have too many still, and the ones that are getting done aren't getting recorded.
<rockstar> :/
<matsubara> why not?
<Ursinha> rockstar, this is a problem
<rockstar> Ursinha, I'll talk with thumper about it.
<Ursinha> rockstar, you'll lost yourselves with that
<Ursinha> rockstar, okay. I'm on my late shift today, will talk to thumper as weel
<Ursinha> *well
<matsubara> rockstar: what's the issue about recording what's been tested?
* salgado is now known as salgado-lunch
<rockstar> matsubara, no idea.  I know some of these things have been tested.
<matsubara> rockstar: right, so it's just a matter of updating the wiki page or something else?
<rockstar> matsubara, wiki page.
<matsubara> I see. if the process to update the test plan page is too troublesome we need to find a better way. suggestions welcome.
<rockstar> matsubara, also, this week has been abnormally busy with the ec2 stuff.  It might be that recording it isn't at the front of their minds.
<rockstar> matsubara, I need to think about that.
<intellectronica> fwiw there's quite a lot of talk about using launchpad itself to track qa. if anyone has good ideas please let them be known
<Ursinha> rockstar, thanks.
<Ursinha> intellectronica, indeed
<rockstar> intellectronica, yea, I was going to think about that.
<rockstar> The Ubuntu guys might be happy to have qa handled in Lanuchpad.
<rockstar> also, Launchpad.
<intellectronica> so i hear, yes
<Ursinha> any suggestions/ideas are very welcome
<matsubara> that's probably something that will happen in the future, until we get to the pojnt we can use LP to track QA, we should be updating the wiki page
<intellectronica> of course
<matsubara> I know that's far from perfect but don't let that stop the QA. 
<matsubara> one thing you could do is email Ursinha and I all items tested and we'll take care of updating the test plan
<Ursinha> and I'm going to repeat myself again, but, please, ping us if we can do something
<Ursinha> like that matsubara said
<matsubara> until we find a better way to deal with this
<bigjools> Ursinha: halp
<matsubara> thanks
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<Ursinha> bigjools, as soon as a have access to mawson, will be glad too. but if there's something I could do now, let me know
<herb> * 2008-12-10 - Cherry picked r7386 enabling replication to production and cherry picked r7419 to edge.
<Ursinha> s/as a/as I/
<herb> * Bugs #118625 and #156453 continue to be a problem despite a patch and the addition of the pyrex package.
<herb> * Had an issue with the branch scanner where a user's subscription to a particularly large branch was causing the branchscanner to hang. See bug #242807
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<herb> * Had to kill off a bunch of old processes on the code hosting server. This seems to be happening a bit more often than once a week now. See bug #260171
<ubottu> Launchpad bug 242807 in ubuntu "my ipod nano is not showing up when plugged in" [Low,Invalid] https://launchpad.net/bugs/242807
<ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
<herb> hmm
<matsubara> ipod nano?
<matsubara> hehe
<Ursinha> lol
<rockstar> 242807 doesn't seem to correlate.
<herb> something tells me I typoed at least one of those.
<herb> should be bug #252807
<ubottu> Bug 252807 on http://launchpad.net/bugs/252807 is private
<rockstar> herb, there are many folken at UDS right now trying to figure out why loggerhead sucks.
<matsubara> herb: btw, did you find out why staging is down?
<rockstar> It's disappointing that it's still having issues.
<matsubara> hmm it's not down anymore, so why staging was down? :-)
<herb> matsubara: the old lp process didn't exit and so the new process couldn't bind to the port after the restore.
<Ursinha> herb, iirc it was the second time this week
<matsubara> herb: ok. thanks for chasing
<Ursinha> the other time happened for the same reason?
<herb> Ursinha: 3rd or 4th time in the last 10 days.
<herb> Ursinha: different reasons the last couple of times. it had been hanging in the middle of startup.
<Ursinha> hm, right.
<rockstar> herb, was it the librarian?
<herb> rockstar: running.  but didn't really look at it in too much detail.
<matsubara> herb: if it happens again, could you file a bug about it please and let me know?
<rockstar> herb, I've noticed the librarian doesn't close cleanly on my dev machine.
<herb> matsubara: sure. I can't provide much detail on the problems I've seen over and over though.  stracing the startup just shows a wait call.
<herb> :(
<matsubara> thanks herb
<matsubara> [TOPIC] * DBA report (DBA contact)
<MootBot> New Topic:  * DBA report (DBA contact)
<stub> The losas altered the production appserver configs so they start talking to the production replica where possible. This is working well... possibly too well.
<stub> Most of the load is now on the slave, with the production box pretty much idling. We can push some of the
<stub> load back to the main box by reverting a few of the appserver configs, but we should probably leave it be until we have the reverse web proxy caching anonymous requests to see how this affects the load distribution.
<stub> I will be running the script to open Jaunty translations with Jeroen tomorrow. This should be the last time it is a heavy weight operation, as translation inheritence is scheduled to be implemented before the next opening. No ETAs on the run time completion yet - I don't fully trust them given the extra overhead we have now on inserts thanks to replication.
<stub> DB patches for this cycle are all either landed or rejected, except for the per language translation policies link with is blocked on discussions upstairs.
<stub> (^-^)
<bigjools> stub: did Celso's get accepted or rejected?
<stub> Which one?
<bigjools> the licensing meta-data one
<stub> Oh... licencing sourcepackagereleae. Rejected. Needs mucho discussion.
<bigjools> blah
<matsubara> all right, anything else for stub?
<matsubara> thanks stub 
<matsubara> anything else before I close?
<matsubara> 5
<matsubara> 4
<matsubara> 3
<matsubara> 2
<matsubara> 1
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
<Ursinha> lol
<matsubara> #endmeeting