1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <sinzui> me
   8 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   9 <Ursinha> me
  10 <herb> me
  11 <intellectronica> me
  12 <matsubara> rockstar, hi
  13 <matsubara> al-maisan, hi
  14 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  15 <matsubara> flacoste, hi
  16 <danilos> me (so-so)
  17 <matsubara> ok, foundations, code and soyuz missing. they can join in later
  18 <matsubara> [TOPIC] Agenda 
  19 <MootBot> New Topic:  Agenda
  20 <matsubara>  * Actions from last meeting
  21 <matsubara>  * Oops report & Critical Bugs 
  22 <matsubara>  * Operations report (mthaddon/herb/spm)
  23 <matsubara>  * DBA report (stub)
  24 <matsubara> [TOPIC] * Actions from last meeting
  25 <MootBot> New Topic:  * Actions from last meeting
  26 <rockstar> me
  27 <cprov> me
  28 <matsubara>     * matsubara to file a bug about the missing select permissions that delayed the rollout
  29 <matsubara>         * https://bugs.edge.launchpad.net/launchpad-foundations/+bug/353926
  30 <matsubara>     * cprov to look up soyuz bugs 353568
  31 <matsubara>     * matsubara to include francis suggestion to bug 353530 and ursinha to summarize what spm told her
  32 <matsubara>         * matsubara commented on the bug.
  33 <matsubara>     * salgado to debug and fix bug 353863
  34 <matsubara>     * sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
  35 <matsubara>     * matsubara to talk to mrevell to announce a maintenance in the DB for about 10 min outage in the next 2 weeks. ask mrevell to talk to stub about it
  36 <matsubara>         * matsubara emailed mrevell about this.
  37 <ubottu> Launchpad bug 353568 in soyuz "ubuntu/source/package/+index timing out" [High,Fix released] https://launchpad.net/bugs/353568
  38 <ubottu> Launchpad bug 353530 in malone "OOPS filing a bug using the email interface " [Undecided,Fix released] https://launchpad.net/bugs/353530
  39 <ubottu> Error: This bug is private
  40 <ubottu> Launchpad bug 353863 in launchpad-registry "TypeError when finishing creating user account in lpnet" [Critical,Fix released] https://launchpad.net/bugs/353863
  41 <sinzui> matsubara: not done
  42 <Ursinha> the info I had was useless to the bug report
  43 <Ursinha> very superficial and not helpful, so I didn't add
  44 <flacoste> me
  45 <matsubara> cprov and salgado bugs are fix released. so that's done
  46 <matsubara> sinzui, do you want me to add another action for your item?
  47 <matsubara> for next week
  48 <sinzui> matsubara: please do
  49 <matsubara> [action] sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
  50 <MootBot> ACTION received:  sinzui to email the list how we should address critical bugs on unmaintained apps (e.g. blueprint)
  51 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  52 <MootBot> New Topic:  * Oops report & Critical Bugs
  53 <matsubara> go ahead Ursinha 
  54 <Ursinha> all right!
  55 <Ursinha> one puzzle for losas/stub, three bugs for foundations, three bugs for registry
  56 <Ursinha> flacoste, bug 354593, bug 353926, openid resetting password, bug 358498
  57 <Ursinha> sinzui: bug 357307, bug 358486, bug 358492
  58 <Ursinha> herb/stub: we're having *lots* of oopses like
  59 <Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1194D1005. I've sent one email to lp list and spoken with jtv about that, something is killing the db connections, so when a request tries to re-use a connection that died it oopses like that. So herb/stub: do you know what can be possibly happening to the db?
  60 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
  61 <ubottu> Bug 353926 on http://launchpad.net/bugs/353926 is private
  62 <ubottu> Launchpad bug 358498 in launchpad-foundations "AssertionError OOPS on openid when resetting password " [Undecided,New] https://launchpad.net/bugs/358498
  63 <ubottu> Launchpad bug 357307 in launchpad-foundations "TypeError when creating new account in lpnet" [Undecided,New] https://launchpad.net/bugs/357307
  64 <ubottu> Launchpad bug 358486 in launchpad-registry "AttributeError when user is confirming new account and LP is checking if it was a suspended one" [Undecided,New] https://launchpad.net/bugs/358486
  65 <ubottu> Launchpad bug 358492 in launchpad-foundations "ProgrammingError OOPS resetting password" [Undecided,New] https://launchpad.net/bugs/358492
  66 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1194D1005
  67 <sinzui> Ursinha: Account and LoginToken are Foundations issues
  68 <herb> Ursinha: looking at the oops now...
  69 <Ursinha> sinzui, looking..
  70 <Ursinha> sinzui, thanks for changing that
  71 <stub> Ursinha: The OOPS is showing that the database reconnection isn't working as it should. Why that connection died isn't on that OOPS - it would have happened on the previous request handled by that thread.
  72 <Ursinha> so that's flacoste's too
  73 * sinzui did it a few minutes ago
  74 <flacoste> sinzui, for AuthToken: +resetpassword, do you think salgado could look into this one?
  75 <sinzui> flacoste: Not this week, He is gone. I can take the  +resetpassword in a few hours
  76 <flacoste> Ursinha: for the branding bug, how often does it happen?
  77 <flacoste> Ursinha: reworking these templates is going to happen, but it's not easy to fix now
  78 <stub> Ursinha: We have watchdogs that kill bad connections. I don't recall seeing any reaped connections from the appservers recently.
  79 <Ursinha> flacoste, about 6,7 a day
  80 <Ursinha> stub, we had 2 thousand oopses like this one I showed
  81 <flacoste> Ursinha: actually, i think these are related to the DisallowedStore error i'm seeing
  82 <flacoste> Ursinha: because these links don't appear on normal SSO pages
  83 <flacoste> stub: what the permission bug in 358492?
  84 <flacoste> nm, i know what this is about
  85 <Ursinha> stub, jtv said he saw a lot of "administrator terminated connection" errors on lp-errors-report list
  86 <stub> When?
  87 <Ursinha> yesterday
  88 <Ursinha> I couldn't find them
  89 <stub> Anyway - the reason the OOPS count is so high is the appserver isn't recovering like it should.
  90 <flacoste> stub: could you look at this bug tomorrow?
  91 <stub> I can look at the oops - I don't know if there is a bug yet (I think there is - not sure though)
  92 <Ursinha> for the db killing spree no, there isn't
  93 <stub> What db killing spree?
  94 <Ursinha> at least I didn't open one, I'll do now
  95 <Ursinha> ah
  96 <Ursinha> I mean, the oopses
  97 <Ursinha> the lots of oopses because of the appserver isn't recovering like it should
  98 <stub> ok
  99 <Ursinha> anyway, it's that bug you're talking about flacoste?
 100 <flacoste> yes
 101 <Ursinha> I'll open one and let you know
 102 <Ursinha> that's all for me
 103 <Ursinha> we have one critical bug, in progress
 104 <Ursinha> so, if matsubara has nothing else to say, oops section is closed
 105 <matsubara> [action] ursinha to file a bug about "appserver isn't recovering like it should causing too many oopses"
 106 <MootBot> ACTION received:  ursinha to file a bug about "appserver isn't recovering like it should causing too many oopses"
 107 <Ursinha> thanks sinzui, flacoste, stub and herb
 108 <Ursinha> and matsubara, of course
 109 <matsubara> intellectronica, can you move bug 269538 from fix committed to fix released?
 110 <ubottu> Launchpad bug 269538 in bugzilla-launchpad/bugzilla-3.2 "Compilation error in plugin when authenticating" [Critical,Fix committed] https://launchpad.net/bugs/269538
 111 <matsubara> or at least chase why it's not fix released yet?
 112 <matsubara> that bug been in fix committed for ages
 113 <intellectronica> matsubara: i have no idea what's going on with that. i'll talk to gmb about it
 114 <matsubara> thanks intellectronica 
 115 <matsubara> thanks everyone
 116 <matsubara> let's move on
 117 <matsubara> [action] intellectronica to talk to gmb about bug 269538
 118 <MootBot> ACTION received:  intellectronica to talk to gmb about bug 269538
 119 <ubottu> Launchpad bug 269538 in bugzilla-launchpad/bugzilla-3.2 "Compilation error in plugin when authenticating" [Critical,Fix committed] https://launchpad.net/bugs/269538
 120 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
 121 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 122 <herb> 2009-04-04 - Launchpad experienced an outage most likely due to hitting some connection limits on the DB. Some users may have experienced issues for up to 90 minutes.
 123 <herb> 2009-04-08 - Deployed r7947 to soyuz and xmlrpc servers.
 124 <herb> Bug 156453 and bug 118625 continue to be problematic for us. Just want to make sure I'm keeping it on your radar.
 125 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
 126 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
 127 <herb> That's all for this week, unless there are questions.
 128 <rockstar> herb, I have something to report here again!
 129 <herb> woohoo!
 130 <rockstar> herb, so we've identified the real memory pig.  Unfortunately, it won't be trivial to change.
 131 <matsubara> cool!
 132 <rockstar> herb, so we know where the issue is, and now we just need to schedule about two weeks and re-write loggerhead.
 133 <herb> haha
 134 <matsubara> hehe
 135 <flacoste> the problem is that he is serious :-/
 136 <herb> mine was a laugh of despair
 137 <matsubara> rockstar, so are you tackling that for 2.2.4 and maybe 2.2.5?
 138 <sinzui> flacoste: I believe the problem with +resetpassword is that it sends logintokens to users who have not setup a person yet.
 139 <rockstar> matsubara, well, I doubt it'll be 2.2.4, because mwhudson is on leave for so much of it.
 140 <rockstar> matsubara, what really needs to happen is that we need to be sequestered again for a week to do nothing but fix it.
 141 <flacoste> sinzui: sounds about right, that shouldn't happen :-)
 142 <sinzui> flacoste: I'll get this fix today
 143 <matsubara> by we, you mean you and mwhudson or the whole code team?
 144 <flacoste> sinzui: thanks a lot
 145 <rockstar> matsubara, mwhudson and I.
 146 <rockstar> matsubara, we got some really good work done at the Pycon sprints last week.
 147 <matsubara> there's all hands and uds coming, maybe during that?
 148 <matsubara> anyway, that's beyond the scope of this meeting.
 149 <matsubara> I think that's all. anything else for herb?
 150 <matsubara> [TOPIC] * DBA report (stub)
 151 <MootBot> New Topic:  * DBA report (stub)
 152 <matsubara> thanks herb 
 153 <stub> Can you describe the memory leak?
 154 <herb> thanks matsubara
 155 <stub> During the last rollout, one of the database patches turned out to be relying on database row ordering for some data migration, with the end result being some newly created rows on the slaves had different primary key values to the master and each other.
 156 <stub> This caused replication to block later when changes to the data on the master could not be duplicated on the slaves due to constraint violations, alerting us to the problem. We rebuild the slave databases to correct the problem (the safest way of recovering the situation).
 157 <stub> The corruption was not noticable to end users and did not infect the master, as only the internal database ids where affected.
 158 <stub> I was hoping to switch our master to the 16 core box, but public holidays and illness have put a hold on that this week.
 159 <stub> On the 6th and 7th, some batch jobs erroneously had their database connections terminated. Sorry about that. It is unlikely this was end user visible.
 160 <stub> echo... echo...
 161 <sinzui> oi oi
 162 <matsubara> stub, you're coordinating the downtime annoucement with mrevell, right?
 163 <stub> I will
 164 <matsubara> stub, ok. thanks.
 165 <matsubara> anything else for stub?
 166 <matsubara> thanks stub.
 167 <matsubara> I think that's all for today.
 168 <matsubara> Thank you all for attending this week's Launchpad Production Meeting.
 169 <matsubara> #endmeeting