1 <MootBot> Meeting started at 10:00. The chair is matsubara.
   2 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   3 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   4 <matsubara> [TOPIC] Roll Call 
   5 <MootBot> New Topic:  Roll Call
   6 <bigjools> me
   7 <gary_poster> me
   8 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   9 <intellectronica> me
  10 <rockstar> ni!
  11 <Ursinha> meee
  12 <matsubara> sinzui, Chex, danilos: hi
  13 <Chex> me
  14 <sinzui> me
  15 <sinzui> matsubara: sorry, I was reading the scrollback
  16 <mbarnett> late me
  17 <danilos> me
  18 <matsubara> stub and danilos can join in later. let's move
  19 <matsubara> [TOPIC] Agenda 
  20 <MootBot> New Topic:  Agenda
  21 <matsubara>  * Actions from last meeting
  22 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  23 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  24 <matsubara>  * DBA report (stub)
  25 <matsubara>  * Proposed items
  26 <matsubara> [TOPIC] * Actions from last meeting
  27 <MootBot> New Topic:  * Actions from last meeting
  28 <matsubara>     * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  29 <matsubara>     * intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
  30 <matsubara>     * ursinha to keep an eye on timeouts for person +branches pages and report back if the CP solved the issue or further action is needed
  31 <matsubara>         * The timeouts are gone
  32 <matsubara>     * matsubara to file a bug for OOPS-1342EA107 (ValueError using the api) and chase someone to fix it
  33 <matsubara>         * Filed bug 423880 and assigned to leonardr
  34 <matsubara>     * ursinha to chase jtv about CP to fix rosetta-poimport script failure
  35 <matsubara>         * The fix is in devel, but no CP will be requested because according to jtv it's unlikely that the problem hits us hard before 3.0. The script stopped failing as well.
  36 <matsubara>     * chex to trawl logs and add request that caused 500 error to bug 422960
  37 <matsubara>     * matsubara to email stub about dba report
  38 <matsubara>         * emailed stub
  39 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  40 <ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
  41 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1342EA107
  42 <ubottu> Launchpad bug 423880 in launchpad-foundations "ValueError: Invalid boundary in multipart form using the API on bug and branch merge proposal objects" [High,Triaged] https://launchpad.net/bugs/423880
  43 <ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,New] https://launchpad.net/bugs/422960
  44 <matsubara> sinzui, I take barry is still busy with 3.0 ui stuff?
  45 <sinzui> matsubara: yes
  46 <matsubara> intellectronica, news about 408738?
  47 <rockstar> EVERYONE is busy with 3.0 UI stuff.
  48 <intellectronica> matsubara: so, we've sceduled the bug for 3.0, and if everything goes according to plan it will be addressed next week. until now we were still very busy with 3.0 UI upgrade
  49 <intellectronica> everyone and their sister, even
  50 <matsubara> all right. I'll keep both action items in. thanks
  51 <matsubara> [action]  barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  52 <MootBot> ACTION received:   barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  53 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  54 <matsubara> [action] intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
  55 <MootBot> ACTION received:  intellectronica to take a look or find someone to take a look on bug 408738 after finishing 3.0 UI stuff
  56 <ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Triaged] https://launchpad.net/bugs/408738
  57 <intellectronica> matsubara: no objection for keeping the action, but mind you it is now scheduled for this milestone
  58 <matsubara> Chex, any news about 422960?
  59 <Ursinha> I wonder if I got disconnected again
  60 <matsubara> nope
  61 <Ursinha> *whew*
  62 <matsubara> well, let's move on then.
  63 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  64 <matsubara> [action]  chex to trawl logs and add request that caused 500 error to bug 422960
  65 <MootBot> ACTION received:   chex to trawl logs and add request that caused 500 error to bug 422960
  66 <ubottu> Launchpad bug 422960 in launchpad-foundations "appear to be failing to record oops for all +translate HTTP 503 errors" [Undecided,New] https://launchpad.net/bugs/422960
  67 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  68 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  69 <Ursinha> me me
  70 <Ursinha> well
  71 <Ursinha> flacoste, stub, I have some oopses like OOPS-1345G2533, that reminded me of bug 174368 (that is fix released), but I couldn't reproduce it
  72 <Ursinha> stub fixed that
  73 <Ursinha> stub, could you take a look, please?
  74 <ubottu> Launchpad bug 174368 in launchpad-foundations "Search query triggering error in tsearch" [Medium,Fix released] https://launchpad.net/bugs/174368
  75 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1345G2533
  76 <Chex> matsubara: I started to look at that last week and was sidetracked, I will still be looking at it
  77 <matsubara> Chex, thanks. I added the action item back. it'll serve as reminder next meeting :-)
  78 <Chex> matsubara: great thanks
  79 <Ursinha> sinzui, also I have OOPS-1348XMLP49, that seems to be related to the problem barry was working on yesterday, the launchpad-reviewers mailing list
  80 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1348XMLP49
  81 <Ursinha> sinzui, do you know if that's correct? if so, is there a bug? (or do you want me to open one?)
  82 <Ursinha> and
  83 <Ursinha> rockstar, bug 427383, can I haz triage in that, please?
  84 <ubottu> Launchpad bug 427383 in launchpad-code "ValueError: No JSON object could be decoded in merge through the API" [Undecided,New] https://launchpad.net/bugs/427383
  85 <sinzui> Ursinha: there is a bug
  86 <gary_poster> Ursinha: you should put me and stub down for that Foundations OOPS
  87 <rockstar> Ursinha, I will look.
  88 <Ursinha> thanks rockstar
  89 <sinzui> Urshina: We do not understand what is wrong and we do not have time to investigate it
  90 <rockstar> Launchpad is teh slow today.
  91 <Ursinha> gary_poster, I'll file a new bug for that and let you know
  92 <gary_poster> Ursinha: thanks
  93 <stub> Ursinha: Yes, it looks similar. Separate bug, same piece of crap code.
  94 <Ursinha> sinzui, I've added a highlight for 'Urshina' just for you here :)
  95 <Ursinha> sinzui, do you know the bug number? I'll keep my eye on that problem and let you know if it gets critical
  96 <stub> Ursinha: Not a high priority though, as it is unlikely to be triggered by anything other than a spam bot (like the OOPS you cited)
  97 <sinzui> Ursinha: bug 403606
  98 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,Triaged] https://launchpad.net/bugs/403606
  99 <Ursinha> sinzui, this bug is the same of that oops? hmm.
 100 * gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has left #launchpad-meeting
 101 * gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has joined #launchpad-meeting
 102 * gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has left #launchpad-meeting
 103 <sinzui> Ursinha: sorry, I was confused..my scrolling is knackered
 104 <Ursinha> :)
 105 <matsubara> [action] ursinha to file a bug for OOPS-1345G2533 and coordinate a fix with gary/stub
 106 <MootBot> ACTION received:  ursinha to file a bug for OOPS-1345G2533 and coordinate a fix with gary/stub
 107 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1345G2533
 108 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1345G2533
 109 <sinzui> Ursinha: That is Barry and LOSAs doing DB surgery.
 110 <intellectronica> siniuz: that's no excuse
 111 <sinzui> Ursinha: That screwed of the data. They fixed it.
 112 <Ursinha> sinzui, all right.
 113 <Ursinha> thanks sinzui
 114 <Ursinha> the only Critical bug is already Fix Committed
 115 * gary_poster (n=anonymou@cpe-069-134-211-019.nc.res.rr.com) has joined #launchpad-meeting
 116 <Ursinha> thanks gary_poster :)
 117 <Ursinha> matsubara, you can move on
 118 <gary_poster> sorry :-P
 119 <Ursinha> thanks guys
 120 <matsubara> all right. thanks Ursinha and everyone
 121 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 122 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 123 <Chex> hello everyone here is what we have for the report:
 124 <Chex> - Buildbot issues in the process of being resolved (currently setting up a staging buildbot)
 125 <Chex> - Higher than normal load on the app servers yesterday which seemed to be caused by higher than normal edge traffic, but not
 126 <Chex> sure if that was slow processing of queries meaning more connections per second, or really more traffic
 127 <Chex> (would need to correlate with apache logs)
 128 <Chex> - Edge outage over the weekend caused by app server hanging during auto edge rollout (bug 307447)
 129 <ubottu> Launchpad bug 307447 in launchpad-foundations "launchpad process not exiting after make stop on staging" [High,In progress] https://launchpad.net/bugs/307447
 130 <Chex> - Codebrowse is still timing out a lot
 131 <Chex> - A few CPs were applied in the past week.
 132 <matsubara> Chex, re: high load on edge, I'll trawl some logs about it today. I'll let you know what I find out
 133 <Chex> and thats it from us. any questions?
 134 <Chex> matsubara: ok, great thank you on the edge load
 135 * sinzui has quit (Read error: 104 (Connection reset by peer))
 136 <gary_poster> 307447 is in progress, btw
 137 <matsubara> Chex, do you know if the LP outage reported by spm might be related with the high load?
 138 <Chex> matsubara: I am not sure, I would have to check on that
 139 * gary_poster notices ubottu beat me to the punch
 140 <matsubara> Chex, ok, we can coordinate after the meeting.
 141 <matsubara> thanks Chex 
 142 <matsubara> [TOPIC] * DBA report (stub)
 143 <MootBot> New Topic:  * DBA report (stub)
 144 <stub> Discussions afoot on migrating to PostgreSQL 8.4. PostgreSQL 8.4.1 has just been released, but will need packaging and backporting to Hardy. We also want Slony-I 1.2.27 to be released and packaged, as that is the version in the 1.2 series that supports PG 8.4 (I think - slony site is having issues right now so I can't fact check).
 145 <stub> Nothing else interesting happening.
 146 <stub> 1.2.17 is the Slony-I release we are waiting on
 147 <stub> (site is back up, or my net connection is)
 148 <matsubara> ok. questions for stub?
 149 <matsubara> guess not. moving on
 150 <matsubara> thanks stub 
 151 <matsubara> [TOPIC] * Proposed items
 152 <MootBot> New Topic:  * Proposed items
 153 <matsubara> no proposed items
 154 * sinzui (n=sinzui@pool-71-178-242-15.washdc.fios.verizon.net) has joined #launchpad-meeting
 155 <matsubara> anything else before I close?
 156 <rockstar> Purple monkey dishwasher.
 157 <matsubara> [action] matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
 158 <MootBot> ACTION received:  matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
 159 <intellectronica> rockstar: i don't understand, what does it wash, dishes or monkeys?
 160 <rockstar> intellectronica, it was a Simpsons reference, and I was just being silly.
 161 <bigjools> or purples
 162 <Ursinha> lol
 163 <Ursinha> gary_poster, stub, bug 427397
 164 <ubottu> Launchpad bug 427397 in launchpad-foundations "Search triggering error in tsearch query again" [Undecided,New] https://launchpad.net/bugs/427397
 165 <Ursinha> matsubara, you can mark that action as completed :P
 166 <gary_poster> thanks Ursinha.  stub, I'm marking it low and throwing it your way.
 167 <matsubara> thanks Ursinha 
 168 <matsubara> ok, I think that's all
 169 <Ursinha> thanks gary_poster
 170 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 171 <Ursinha> and matsubara
 172 <Ursinha> and stub
 173 <matsubara> #endmeeting
 174 <MootBot> Meeting finished at 10:26.