## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 09:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
<matsubara> [TOPIC] Roll Call
<MootBot> New Topic:  Roll Call
<sinzui> me
<gmb> me
<gmb> (in for intellectronica)
* flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
<henninge_> me
<flacoste> me
<bigjools> me
* Ursinha (n=ursula@canonical/launchpad/ursinha) has joined #launchpad-meeting
<Ursinha> erm
<Ursinha> me?
<mrevell> me
<matsubara> stub, herb, henninge?
<matsubara> rockstar: ?
<stub> me
<henninge_> me!
<herb> me
<rockstar> me
<rockstar> Sorry, was ill yesterday, still not at 1[[%
<rockstar> 100%
<matsubara> all right, everyone is here
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Next meeting
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs 
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (DBA contact)
<matsubara> so next meeting, same time next week?
<matsubara> I guess that's a yes
<matsubara> all right
<matsubara> [topic] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * rockstar to work with beuno re: critical loggerhead (bug 156453)
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<matsubara> rockstar: any news about that one?
<rockstar> matsubara, working on it.
<Ursinha> +1
<rockstar> I don't think it's something that's going to happen overnight.
<Ursinha> rockstar, is it really critical?
<rockstar> Ursinha, well, yes.
* henninge has quit (Read error: 110 (Connection timed out))
<matsubara> rockstar: any eta?
<rockstar> I bet the LOSAs think it's psat Critical.
* henninge_ is now known as henninge
<henninge> me again
<rockstar> matsubara, until we're sure what the problem is, I can't give you an ETA.
<matsubara> rockstar: or can you give us regular status report on the bug report itself? critical bugs shouldn't be left like nobody is looking at them
* herb definitely thinks it's critical.
<herb> particularly since I got a page about it right before the meeting. :)
<rockstar> matsubara, I don't really have much to report on it.
<rockstar> It's easy to make it leak memory, it's hard to see where it is leaking memory.
<Ursinha> rockstar, nothing, like partial reports, what are you guys thinking to do
<Ursinha> ?
<Ursinha> oh
<Ursinha> see
<matsubara> rockstar: by looking at the report it seems that it's been stuck since 2008-11-06
<rockstar> matsubara, I can put "Still working on this" every once in a while.  I just don't have specifics, unfortunately.
<matsubara> right
<rockstar> Okay, I'll do that.
<matsubara> thanks rockstar 
<matsubara> [TOPIC]  * Oops report & Critical Bugs 
<MootBot> New Topic:   * Oops report & Critical Bugs
* matsubara hands the mic to Ursinha 
<Ursinha> ok
<Ursinha> mics make me nervous
* bigjools waits for the Karaoke
<Ursinha> have two bugs and one timeout
<Ursinha> matsubara's bug: 300320
<Ursinha> argh
<Ursinha> bug 300320
<ubottu> Launchpad bug 300320 in launchpad-foundations "mailman oops directory permission keep getting reset" [High,New] https://launchpad.net/bugs/300320
<matsubara> that's not critical but it's blocking our work
<Ursinha> so the bot isn't here
<matsubara> I've asked herb to fix the permission
<Ursinha> hahaha
<Ursinha> ok
<Ursinha> so matsubara filed this one
<matsubara> but we need to find out what's resetting the permission every once in a while
<herb> matsubara: trying to do some debugging on that, but will reset the permissions after the meeting.
<Ursinha> matsubara, indeed, just got one of these before the meeting
<matsubara> thanks herb. let me know any news on the report please
<matsubara> flacoste: I'm not sure that's a foundations or registry issue, btw 
<Ursinha> ok
<sinzui> Bug 300183 is critical. I just made it critical
<ubottu> Bug 300183 on http://launchpad.net/bugs/300183 is private
<flacoste> matsubara: registry
<Ursinha> the other is for bugs, bug 300324
<ubottu> Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324
* Ursinha kicks ubottu 
<flacoste> sinzui: you are aware of bug 300320?
<ubottu> Launchpad bug 300320 in launchpad-foundations "mailman oops directory permission keep getting reset" [High,New] https://launchpad.net/bugs/300320
<matsubara> flacoste: I'll re-assign and coordinate with barry and sinzui then if it's a code or LP config problem. thanks
<Ursinha> it's the bug regarding that bug that had its comment 0 deleted accidentally
<gmb> Ursinha: I've just triaged that one.
<gmb> The simple fix is to delete all of bug 297388's subscribers
<ubottu> Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text)
<gmb> and to mark it private
<gmb> hush, ubottu
<sinzui> flacoste: I saw it today. I believed it related install problem that is bug 300183
<ubottu> Bug 300183 on http://launchpad.net/bugs/300183 is private
<gmb> Doing that will make it invisible, to all intents and purposes.
<stub> The OOPS dir permissions are probably being reset by rsync, so the fix is to ensure permissions are set correctly on forster or to tell rsync to make everything locally world readable
<Ursinha> gmb, thanks, it's really annoying and I'm just waiting for some nice bugs guy to solve that for us
<gmb> Ursinha: Ultimately we want to fix bug 297411
<ubottu> Launchpad bug 297411 in malone "Delete a given bug" [Undecided,Triaged] https://launchpad.net/bugs/297411
<Ursinha> hahaha you just caused an oops
<gmb> But it's not easy to do.
* Ursinha kicks ubottu again
<matsubara> thanks stub. 
<Ursinha> gmb, I saw BjornT 's comment
<Ursinha> gmb, but for now we need something to stop this bug to oops
<stub> gmb: If we only hit the red button for spam, we can do it by changing a load of constraints to ON DELETE CASCADE.
<Ursinha> it's filling the summaries with oopses
<gmb> Ursinha: Well, hiding it should do that.
<gmb> Because veeryone will get "you can't access this bug" eros.
<gmb> Erk.
<gmb> Waaaay wrong word, there.
<gmb> *errors
<Ursinha> gmb, hm
<Ursinha> gmb, is it the easiest thing to do?
<gmb> Ursinha: Yes. By far.
<gmb> The trouble is that the Bug table has a lot of dependencies
<gmb> And the dependencies also have a lot of dependencies
<gmb> So we have to find all the things that might depend upon the Bug record before we can DELETE it.
<Ursinha> gmb, well, but do you need that when hiding?
<Ursinha> considering hiding != deleting
<gmb> UPDATE Bug SET private = True WHERE id = 287411;
<gmb> DELETE FROM BugSubscription WHERE bug = 297411;
* Notify: kiko is offline (lem.freenode.net).
<gmb> (yes, the first number was wrong)
<gmb> And that should do it.
<Ursinha> gmb, making it private will prevent privileged people to get the oops?
<matsubara> gmb: can you add that to LaunchpadProductionStatus page and ask the Losas to run it?
<gmb> Good question.
<gmb> matsubara: Sure.
<stub> gmb: But we can ignore a lot of the dependencies - if this is spam protection, those dependencies won't exist.
<matsubara> [action] gmb to add sql to hide bug 297411 as a fix for bug 300324
<MootBot> ACTION received:  gmb to add sql to hide bug 297411 as a fix for bug 300324
<ubottu> Launchpad bug 297411 in malone "Delete a given bug" [Undecided,Triaged] https://launchpad.net/bugs/297411
<ubottu> Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324
<gmb> stub: Is there a way to do a DELETE without Postgres erroring on un-deleted dependenices?
<Ursinha> another oops coming..
<matsubara> shit, that action is wrong
<matsubara> [action] gmb to add sql to hide bug 297388 as a fix for bug 300324
<MootBot> ACTION received:  gmb to add sql to hide bug 297388 as a fix for bug 300324
<gmb> Ursinha: If ubottu listens to its own drivel.
<ubottu> Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text)
<ubottu> Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text)
<gmb> Damn.
<stub> gmb: No, but we can put up with an oops in that case since an admin is hitting the button when they shouldn't
<Ursinha> arrrrr
<gmb> stub: I'm not quite sure that I'm on your wavelength here. Are you talking about a fix for the OOPS Ursinha raised or for bug 297411 in general?
<ubottu> Launchpad bug 297411 in malone "Delete a given bug" [Undecided,Triaged] https://launchpad.net/bugs/297411
* gmb avoids another OOPS
<stub> gmb: Bug deletion in general (so wrong channel I guess)
<matsubara> Ursinha: anything else?
<gmb> stub: Ah right. Yes, let's talk about that elsewhere and not pollute the meeting.
<Ursinha> thanks gmb :P
<matsubara> I think we can discuss the bug deletion bug in the mailing list or whatever. it's not a simple fix anyway
<Ursinha> matsubara, I have one timeout oops
<Ursinha> OOPS-1055EC45
<Ursinha> I'd like someone to take a look on it
<Ursinha> few occurrences but high sql time, after the rollout
<Ursinha> ubottu, I hate you man
<ubottu> Sorry, I don't know anything about i hate you man
<Ursinha> lol
<Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1055EC45
<matsubara> sinzui: ^
<matsubara> can you take a look?
<Ursinha> I have an incredible lag today
<sinzui> yuck
<flacoste> matsubara: could you make the left pane on that page viewport fixed (so that it is always available wherever you are on the oops)?
<flacoste> matsubara: the oops report page i mean
<Ursinha> flacoste, good idea
<matsubara> flacoste: yeah, got that. will do
<matsubara> flacoste: and thanks for the suggestion :-)
<matsubara> [action] ursinha to file a bug about OOPS-1055EC45 and coordinate with Registry to get it fixed
<Ursinha> this is all from me
<MootBot> ACTION received:  ursinha to file a bug about OOPS-1055EC45 and coordinate with Registry to get it fixed
<matsubara> let's move on then
<matsubara> thanks Ursinha and everyone else
<matsubara> [topic] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<sinzui> matsubara: flacoste I can pass this to Edwin because I think he has  day for this.
<herb>  2008-11-19: Rolled out r7316 to production.
<herb> - Bugs #118625 and #156453 continue to be a problem causing us to restart codebrowse on a number of ocassions this week. As we discussed earlier.
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<herb> - We can probably mark bug #247227 as Fix Released since the app servers have been stable for several weeks now.
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<ubottu> Bug 247227 on http://launchpad.net/bugs/247227 is private
<herb> - Do we have a decision on if/when there will be a re-roll?
<Ursinha> thanks guys
<matsubara> thanks sinzui 
<matsubara> Ursinha: add that the report ^ :-)
<matsubara> s/the/to the/
<Ursinha> matsubara, will do
<matsubara> herb: I talk to kiko about it today and let you know
<matsubara> about the re-roll I mean
<herb> matsubara: ok
<Ursinha> I'll ask him to join
* kiko-fud (n=kiko@canonical/launchpad/kiko) has joined #launchpad-meeting
<kiko-fud> yes dears
<Ursinha> <herb> - Do we have a decision on if/when there will be a re-roll?
<kiko-fud> tonight 22:00 UTC if at all
<kiko-fud> I have spoken
<matsubara> cool. thanks kiko-fud 
<Ursinha> ok, great
<matsubara> thanks herb 
<Ursinha> thanks kiko-fud
<matsubara> [topic] * DBA report (DBA contact)
<MootBot> New Topic:  * DBA report (DBA contact)
<herb> thanks Ursinha, matsubara, kiko-fud
<herb> et al
* matsubara hands the mic to stub 
<stub> Our replica production database built happily, and maintaining the replica does not seem to be adding noticable load. My worries about diskspace where thankfully unfounded.
<stub> We may find that chokecherry's load is too high when we start the appservers using the replica during staging rebuilds. If this turns out to be the case, we have the choice of not using the replica or moving staging somewhere else.
<stub> oot
<matsubara> stub: and the plan is to have that on production next cycle?
<stub> Its just a config change - we can switch it on whenever.
<matsubara> cool.
<stub> I've got a branch prepared with the config changes and kiko knows already
<matsubara> Ursinha: ^ got your answer for waht you asked me yesterday? :-0
<matsubara> :-)
<matsubara> great
<matsubara> thanks stub 
<Ursinha> matsubara, yes :)
<matsubara> anything else before I close?
<flacoste> matsubara: i'll land that branch
<Ursinha> thanks matsubara and statik
<Ursinha> errr
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
<matsubara> #endmeeting 
<Ursinha> and stub
<Ursinha> sorry statik