## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<bigjools> me
<gary_poster> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<intellectronica> me
<rockstar> ni!
<Ursinha> meee
<matsubara> sinzui, Chex, danilos: hi
<Chex> me
<sinzui> me
<sinzui> matsubara: sorry, I was reading the scrollback
<mbarnett> late me
<danilos> me
<matsubara> stub and danilos can join in later. let's move
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>     * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<matsubara>     * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
<matsubara>     * ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
<matsubara>         * The timeouts are gone
<matsubara>     * matsubara to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
<matsubara>         * Filed bug 423880 and assigned to leonardr
<matsubara>     * ursinha to chase jtv about CP to fix rosetta-poimport script failure
<matsubara>         * The fix is in devel, but no CP will be requested because according to jtv it's unlikely that the problem hits us hard before 3.0. The script stopped failing as well.
<matsubara>     * chex to trawl logs and add request that caused 500 error to bug 422960
<matsubara>     * matsubara to email stub about dba report
<matsubara>         * emailed stub
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107
<ubottu> Launchpad bug 423880 in launchpad-foundations "ValueError: Invalid boundary in multipart form using the API on bug and branch merge proposal objects" [High,Triaged] https://launchpad.net/bugs/423880
<ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,New] https://launchpad.net/bugs/422960
<matsubara> sinzui, I take barry is still busy with 3.0 ui stuff?
<sinzui> matsubara: yes
<matsubara> intellectronica, news about 408738?
<rockstar> EVERYONE is busy with 3.0 UI stuff.
<intellectronica> matsubara: so, we've sceduled the bug for 3.0, and if everything goes according to plan it will be addressed next week. until now we were still very busy with 3.0 UI upgrade
<intellectronica> everyone and their sister, even
<matsubara> all right. I'll keep both action items in. thanks
<matsubara> [action]  barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<MootBot> ACTION received:   barry to continue debug on bug 403606 after finishing 3.0 UI stuff
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<matsubara> [action] intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
<MootBot> ACTION received:  intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
<ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
<intellectronica> matsubara: no objection for keeping the action, but mind you it is now scheduled for this milestone
<matsubara> Chex, any news about 422960?
<Ursinha> I wonder if I got disconnected again
<matsubara> nope
<Ursinha> *whew*
<matsubara> well, let's move on then.
* stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
<matsubara> [action]  chex to trawl logs and add request that caused 500 error to bug 422960
<MootBot> ACTION received:   chex to trawl logs and add request that caused 500 error to bug 422960
<ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,New] https://launchpad.net/bugs/422960
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> me me
<Ursinha> well
<Ursinha> flacoste, stub, I have some oopses like OOPS-1345G2533, that reminded me of bug 174368 (that is fix released), but I couldn't reproduce it
<Ursinha> stub fixed that
<Ursinha> stub, could you take a look, please?
<ubottu> Launchpad bug 174368 in launchpad-foundations "Search query triggering error in tsearch" [Medium,Fix released] https://launchpad.net/bugs/174368
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1345G2533
<Chex> matsubara: I started to look at that last week and was sidetracked, I will still be looking at it
<matsubara> Chex, thanks. I added the action item back. it'll serve as reminder next meeting :-)
<Chex> matsubara: great thanks
<Ursinha> sinzui, also I have OOPS-1348XMLP49, that seems to be related to the problem barry was working on yesterday, the launchpad-reviewers mailing list
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1348XMLP49
<Ursinha> sinzui, do you know if that's correct? if so, is there a bug? (or do you want me to open one?)
<Ursinha> and
<Ursinha> rockstar, bug 427383, can I haz triage in that, please?
<ubottu> Launchpad bug 427383 in launchpad-code "ValueError: No JSON object could be decoded in merge through the API" [Undecided,New] https://launchpad.net/bugs/427383
<sinzui> Ursinha: there is a bug
<gary_poster> Ursinha: you should put me and stub down for that Foundations OOPS
<rockstar> Ursinha, I will look.
<Ursinha> thanks rockstar
<sinzui> Urshina: We do not understand what is wrong and we do not have time to investigate it
<rockstar> Launchpad is teh slow today.
<Ursinha> gary_poster, I'll file a new bug for that and let you know
<gary_poster> Ursinha: thanks
<stub> Ursinha: Yes, it looks similar. Separate bug, same piece of crap code.
<Ursinha> sinzui, I've added a highlight for 'Urshina' just for you here :)
<Ursinha> sinzui, do you know the bug number? I'll keep my eye on that problem and let you know if it gets critical
<stub> Ursinha: Not a high priority though, as it is unlikely to be triggered by anything other than a spam bot (like the OOPS you cited)
<sinzui> Ursinha: bug 403606
<ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
<Ursinha> sinzui, this bug is the same of that oops? hmm.
* gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has left #launchpad-meeting
* gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has joined #launchpad-meeting
* gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has left #launchpad-meeting
<sinzui> Ursinha: sorry, I was confused..my scrolling is knackered
<Ursinha> :)
<matsubara> [action] ursinha to file a bug for OOPS-1345G2533 and coordinate a fix with gary/stub
<MootBot> ACTION received:  ursinha to file a bug for OOPS-1345G2533 and coordinate a fix with gary/stub
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1345G2533
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1345G2533
<sinzui> Ursinha: That is Barry and LOSAs doing DB surgery.
<intellectronica> siniuz: that's no excuse
<sinzui> Ursinha: That screwed of the data. They fixed it.
<Ursinha> sinzui, all right.
<Ursinha> thanks sinzui
<Ursinha> the only Critical bug is already Fix Committed
* gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has joined #launchpad-meeting
<Ursinha> thanks gary_poster :)
<Ursinha> matsubara, you can move on
<gary_poster> sorry :-P
<Ursinha> thanks guys
<matsubara> all right. thanks Ursinha and everyone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Chex> hello everyone here is what we have for the report:
<Chex> - Buildbot issues in the process of being resolved (currently setting up a staging buildbot)
<Chex> - Higher than normal load on the app servers yesterday which seemed to be caused by higher than normal edge traffic, but not
<Chex> sure if that was slow processing of queries meaning more connections per second, or really more traffic
<Chex> (would need to correlate with apache logs)
<Chex> - Edge outage over the weekend caused by app server hanging during auto edge rollout (bug 307447)
<ubottu> Launchpad bug 307447 in launchpad-foundations "launchpad process not exiting after make stop on staging" [High,In progress] https://launchpad.net/bugs/307447
<Chex> - Codebrowse is still timing out a lot
<Chex> - A few CPs were applied in the past week.
<matsubara> Chex, re: high load on edge, I'll trawl some logs about it today. I'll let you know what I find out
<Chex> and thats it from us. any questions?
<Chex> matsubara: ok, great thank you on the edge load
* sinzui has quit (Read error: 104 (Connection reset by peer))
<gary_poster> 307447 is in progress, btw
<matsubara> Chex, do you know if the LP outage reported by spm might be related with the high load?
<Chex> matsubara: I am not sure, I would have to check on that
* gary_poster notices ubottu beat me to the punch
<matsubara> Chex, ok, we can coordinate after the meeting.
<matsubara> thanks Chex 
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> Discussions afoot on migrating to PostgreSQL 8.4. PostgreSQL 8.4.1 has just been released, but will need packaging and backporting to Hardy. We also want Slony-I 1.2.27 to be released and packaged, as that is the version in the 1.2 series that supports PG 8.4 (I think - slony site is having issues right now so I can't fact check).
<stub> Nothing else interesting happening.
<stub> 1.2.17 is the Slony-I release we are waiting on
<stub> (site is back up, or my net connection is)
<matsubara> ok. questions for stub?
<matsubara> guess not. moving on
<matsubara> thanks stub 
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no proposed items
* sinzui (n=sinzui@pool-71-178-242-15.washdc.fios.verizon.net) has joined #launchpad-meeting
<matsubara> anything else before I close?
<rockstar> Purple monkey dishwasher.
<matsubara> [action] matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<MootBot> ACTION received:  matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<intellectronica> rockstar: i don't understand, what does it wash, dishes or monkeys?
<rockstar> intellectronica, it was a Simpsons reference, and I was just being silly.
<bigjools> or purples
<Ursinha> lol
<Ursinha> gary_poster, stub, bug 427397
<ubottu> Launchpad bug 427397 in launchpad-foundations "Search triggering error in tsearch query again" [Undecided,New] https://launchpad.net/bugs/427397
<Ursinha> matsubara, you can mark that action as completed :P
<gary_poster> thanks Ursinha.  stub, I'm marking it low and throwing it your way.
<matsubara> thanks Ursinha 
<matsubara> ok, I think that's all
<Ursinha> thanks gary_poster
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<Ursinha> and matsubara
<Ursinha> and stub
<matsubara> #endmeeting
<MootBot> Meeting finished at 10:26.