#format IRC
<Ursinha> #startmeeting
<Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<MootBot> Meeting started at 11:02. The chair is Ursinha.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<rockstar> "Gimp's sleepin'" "Well wake him up"
<Ursinha> [TOPIC] Roll Call
<Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<MootBot> New Topic:  Roll Call
<sinzui> me
* rockstar  
<Ursinha> me
<Ursinha> and all the others that already meed
<Ursinha> stub is excused
<Ursinha> gary_poster, allenap, hello
<allenap> me
<danilos> me (maybe I get counted twice)
<gary_poster> Ursinha: on call but sort of here
<Ursinha> :)
<Ursinha> gary_poster, ok, maybe matsubara can join us in your behalf?
<Ursinha> I'm moving on, will poke you when it's time :)
<Ursinha> [TOPIC] Agenda
<Ursinha>  * Actions from last meeting
<Ursinha>  * Oops report & Critical Bugs & Broken scripts
<Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
<Ursinha>  * DBA report (stub)
<Ursinha>  * QA stats of the week
<Ursinha>  * Proposed items
<MootBot> New Topic:  Agenda
<Ursinha> [TOPIC] * Actions from last meeting
<Ursinha> * matsubara to file a bug for OOPS-1553EC1332
<Ursinha>     * Filed https://bugs.edge.launchpad.net/launchpad-code/+bug/558597
<Ursinha> * matsubara to chase critical bug 556839 with noodles or someone from soyuz
<Ursinha>     * chatted with noodles about this one. Jelmer is investigating.
<Ursinha> * matsubara chase someone from soyuz about expire-ppa-files failing script
<Ursinha>     * chatted with noodles about this one and he's going to reply to the failure email
<Ursinha> * gary_poster to investigate OOPS-1557EA572 and re-open or file a new bug for it
<Ursinha>     * Not needed. it's a code issue (bug 558585)
<Ursinha> * matsubara to file a bug on launchpad-code about OOPS-1557EA572 including
<Ursinha>   details from salgado ("the OOPS above is different from the one I'd seen when
<Ursinha>   fixing the bug.  that's in the view that generates merge-proposal diffs.  
<Ursinha>   that view should either use StreamOrRedirectLibraryFileAliasView (which knows
<Ursinha>   what to do when the librarian is down) or do something similar to that")
<Ursinha>     * Done: https://bugs.edge.launchpad.net/launchpad-code/+bug/558585
<Ursinha> * matsubara to talk to foundations about launchpadlib untriaged items
<Ursinha>     * Done: talked to Gary about this and triaged a few bugs.
<MootBot> New Topic:  * Actions from last meeting
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1553EC1332
<ubottu> Launchpad bug 558597 in launchpad-code "OOPS using person branch feed" [High,Triaged]
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556839)
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1557EA572
<ubottu> Launchpad bug 558585 in launchpad-code "branch merge proposal diff page should handle librarian outage better" [High,Triaged] https://launchpad.net/bugs/558585
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1557EA572
<Ursinha> it seems all bugs were filed and oopses sorted out
<ubottu> Launchpad bug 558585 in launchpad-code "branch merge proposal diff page should handle librarian outage better" [High,Triaged]
<Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> all relevant bugs were discussed with people during this week; the most offenders are bug 553361 (~1200 oopses a day, targetted to 10.04) and bug 54005 (we're not sure this is the real one, given it's old)
<Ursinha> bug 54005 (OOPS-1564CEMAIL1) is hard to debug given the almost empty oops report, but this emptyness is bug 230106, which deryck said won't be fixed now but discussed soon
<Ursinha> we're having other email related oops, besides the OOPS-1564CEMAIL1 one: OOPS-1563CEMAIL101, to which I filed bug 563055, and gary_poster already triaged them
<Ursinha> sinzui, do you know what's going on in this oops? OOPS-1564B963
<ubottu> Launchpad bug 553361 in launchpad-foundations "OOPS when accessing launchpad with no referrer, eg. via wget" [High,Triaged] https://launchpad.net/bugs/553361
<ubottu> Launchpad bug 54005 in launchpad-foundations "bugs.launchpad.ubuntu.com email ending up in Launchpad mailbox" [Low,Triaged] https://launchpad.net/bugs/54005
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1564CEMAIL1
<ubottu> Launchpad bug 563055 in launchpad-foundations "Mail parsing failing with ValueError" [Low,Triaged] https://launchpad.net/bugs/563055
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1564CEMAIL1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1563CEMAIL101
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1564B963
<Ursinha> sinzui, hi :)
<sinzui> Ursinha, it looks like an insane file name
* matsubara (~matsubara@canonical/launchpad/matsubara) has joined #launchpad-meeting
<Ursinha> sinzui, is it possible to upload files with insane names?
<sinzui> I do not know
<sinzui> nunitv2
<sinzui> has one it seems
<allenap> Ursinha: I've now also commented on bug 230106, https://bugs.edge.launchpad.net/malone/+bug/230106/comments/3
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged]
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
* sinzui looks on site
<Ursinha> thanks allenap 
<Ursinha> there's also another oops that I'm not sure who should take a look
<Ursinha> it's OOPS-1565O1873
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1565O1873
<Ursinha> maybe sinzui or gary_poster 
<gary_poster> looking
<sinzui> Ursinha, the file is sane
<sinzui> http://edge.launchpad.net/nunitv2/2.5/2.5.3/+download/NUnit-2.5.3.9345.msi
<MootBot> LINK received:  http://edge.launchpad.net/nunitv2/2.5/2.5.3/+download/NUnit-2.5.3.9345.msi
<Ursinha> sinzui, I saw that
<gary_poster> Ursinha, I'll throw it to sinzui unless he throws it back.  we generally haven't worked on the gog stuff to my knowledge
<gary_poster> gpg
<Ursinha> I just didn't get why that happened
<Ursinha> allenap, your comment is interesting, would that be too hard to do?
<allenap> Ursinha: No, not too hard. Abel has made the statement logging code for webapps work well in a longer-running script context.
<sinzui> gary_poster, I accepted the bug, but the issue still looks to be an issue between slave and master
<Ursinha> allenap, cool! could you please check with deryck what does he thing about that idea? :)
<allenap> Ursinha: Okay.
<Ursinha> allenap, thanks
<Ursinha> sinzui, what should we do about that?
<sinzui> I do not know
<sinzui> I saw this happen in the UI two weeks ago
<sinzui> The CoC oopsed for 10 minutes then worked when the key was really visible
<Ursinha> hm
<Ursinha> sinzui, should I open a bug so that can be investigated properly?
<gary_poster> sinzui: re bug that you accepted: would you bring it up with me out of meeting then?  I'm happy to help, or take it from you if appropriate, but my understanding is that the answer is to use the context manager I mentioned before, and that you chould just use it to always connect to the master for this purpose.  I May be off base, so I'd like to coordinate with you 
.
<sinzui> We have a fix for the page to step out of that condition, but I do not know how to make the master pass the data appear in a timely fashion
<gary_poster> sinzui: do you mean you need info on the context manager I'm mentioning.  can find
<Ursinha> gary_poster, sinzui, I'll file a bug for that, is that ok for you guys?
<sinzui> These is a bug
<gary_poster> Yeah thought so
<sinzui> bug 562486
<ubottu> Launchpad bug 562486 in launchpad-registry "accessing pending_gpg_keys using the api fails to some users" [High,Triaged] https://launchpad.net/bugs/562486
<Ursinha> sinzui, okay, thanks
<Ursinha> [action] gary_poster and sinzui to discuss about the issues between master and slave that could be causing oopses
<MootBot> ACTION received:  gary_poster and sinzui to discuss about the issues between master and slave that could be causing oopses
<Ursinha> [action] allenap to follow up with deryck about bug 230106
<MootBot> ACTION received:  allenap to follow up with deryck about bug 230106
<ubottu> Launchpad bug 230106 in malone "emails interface oops reports need better error messages" [Low,Triaged] https://launchpad.net/bugs/230106
<Ursinha> I'm moving on 
<Ursinha> we have three critical bugs, the two foundations ones are fix committed, one soyuz is in progress
<Ursinha> jelmer is working on the soyuz one
<Ursinha> I guess bug 514267 and bug 530002 are actually Fix Released, right gary_poster?
<ubottu> Launchpad bug 514267 in launchpad-foundations "InternalError on clusters under busy load" [Critical,Fix committed] https://launchpad.net/bugs/514267
<ubottu> Launchpad bug 530002 in launchpad-foundations "Launchpad edge 'devel' webservice serving 'beta' WADL" [Critical,Fix committed] https://launchpad.net/bugs/530002
<sinzui> Ursinha, The filename oops was caused when a chinese user used the chinese name to access a file that does not exist
<Ursinha> sinzui, oh boy
<Ursinha> chinese name
<sinzui> I think there was a transliteration problem for the user.
<gary_poster> yes, and...
<Ursinha> hehe
<Ursinha> sinzui, you mean he typed the url by hand?
<gary_poster> yes
<sinzui> I think so
<gary_poster> both are now marked as resolved
<Ursinha> thanks gary_poster 
<Ursinha> sinzui, well, so I guess there's almost nothing we could do about that..
<Ursinha> in the failing scripts land, a fix for retry-depwait has landed production-devel, so the only script constantly failing has been fixed, it seems
<Ursinha> thanks everyone
<Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Ursinha> hi Chex 
* Ursinha hands Chex the mic
<Ursinha> or the keyboard :)
* Ursinha pokes Chex in every possible channel
<Chex> sorry
<bigjools> oouch!
<Chex> LP report for this week is short:
<Ursinha> SNAFU? :P
<Chex> - LP incidents of note:
<Chex>         : 11-Apr: swap alerts. restarted some LPNet servers. Bug: 531071
<Chex>         : 15-Apr: Librarian1 rapidly gobbling memory again, restarted to free up.  Bug 556245.
<ubottu> Launchpad bug 531071 in launchpad-foundations "lpnet app servers are leaking memory again" [High,Triaged] https://launchpad.net/bugs/531071
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<rockstar> Looks like Launchpad is timing out for ubottu as well as myself.
<gary_poster> that's actually a different one
<gary_poster> you mean...
<Chex> these are existing issues, but I just bring them up as they occurred in the last week. any questions or comments?
<gary_poster> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/556245
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<gary_poster> oh
<gary_poster> duh
* gary_poster grins shamefacedly
<Ursinha> rockstar, that's a private bug, maybe that's why?
<gary_poster> yeah prob
<rockstar> Ursinha, ah, maybe.
<rockstar> I've been getting lots of timeouts on Launchpad recently...
<Ursinha> rockstar, any specific pages?
<rockstar> Ursinha, mostly bugs stuff.
<Ursinha> well, lp is working pretty fine for me these days, I'm actually happy with that :)
<Ursinha> rockstar, could you let me know when that happens, please?
<Ursinha> and about Chex bugs, anyone has something to add?
<bigjools> lp is sloooow right now
<rockstar> bigjools, thank you for confirming what I just said.
<bigjools> it just took about 13 seconds to load a simple page
<Ursinha> ouch
<Ursinha> Chex, do you know if is there anything going on right now that could be causing that?
<Ursinha> bigjools, do you have a link so I can test it here, please?
<bigjools> https://edge.launchpad.net/~launchpad/+archive/ppa/+build/1693771
<Chex> Ursinha: I just quickly looked at the lpnet servers, and db-master, I dont see anything striking at the moment
<Ursinha> bigjools, yes, that's really slow
<Ursinha> I'll check the timeouts more closely
<Ursinha> thanks bigjools, rockstar and Chex 
<Chex> Ursinha: welcome
<Ursinha> gary_poster, about that bug Chex mentioned, is there anything that would be done soon?
<Ursinha> bug 556245 
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<Ursinha> I guess not
<Ursinha> I'll move on to finish this meeting today :)
<gary_poster> Ursinha: I'm going to ask stub to investigate it.  That's all I can say for now.
<Ursinha> gary_poster, that's fine, thanks
<Ursinha> [action] gary_poster to ask stub to investigate bug 556245
<MootBot> ACTION received:  gary_poster to ask stub to investigate bug 556245
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<ubottu> Error: Could not parse data returned by Launchpad: list index out of range (https://launchpad.net/bugs/556245)
<Ursinha> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<Ursinha> [action] Ursinha to send an email to stub asking for the dba report
<MootBot> ACTION received:  Ursinha to send an email to stub asking for the dba report
<Ursinha> [TOPIC] * QA stats of the week
<MootBot> New Topic:  * QA stats of the week
<Ursinha> and here are the untriaged bugs of this week
<Ursinha> launchpadlib: 13
<Ursinha> lp-dev-utils: 9
<Ursinha> launchpad-buildd: 9
<Ursinha> launchpad-foundations: 6
<Ursinha> launchpad-dev-moin-theme: 5
<Ursinha> lp-qa-tools: 3
<Ursinha> lpbuildbot: 2
<Ursinha> trac-launchpad-migrator: 2
<Ursinha> launchpad-loggerhead: 2
<Ursinha> launchpad-news-wordpress-theme: 2
<Ursinha> launchpad-help-moin-theme: 2
<Ursinha> launchpad-cscvs: 2
<Ursinha> launchpad-registry: 1
<Ursinha> launchpad: 1
<Ursinha> launchpad-documentation: 1
<Ursinha> tickcount: 1
<Ursinha> malone: 1
<Ursinha> so, it seems soyuz managed to triage of their bugs :)
<bigjools> \o/ \o/ \o/
* rockstar gives Soyuz a cookie
<Ursinha> bigjools, congratulations :)
<gary_poster> :-)
<bigjools> did the last 30 this morning :)
<rockstar> You have to split between yourselves though.  I only had one cookie.
<Ursinha> haha
<Ursinha> :))
<Ursinha> I'll talk to people about the other ones
<Ursinha> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<Ursinha> anyone have anything else to add?
<rockstar> I have one thing to say, which I'll express in a haiku (ala barry)
<Ursinha> go ahead 
<rockstar> Code team sprinting / this meeting is at 4 am / I am not waking up
<Ursinha> lol
<rockstar> That's for next week.
<Ursinha> thanks for letting me know
* Ursinha gives rockstar a cookie for his creativity
<Ursinha> so I guess that's all for this week
<Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
<Ursinha> #endmeeting