1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 * flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
   5 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   6 <matsubara> [TOPIC] Roll Call 
   7 <MootBot> New Topic:  Roll Call
   8 <flacoste> me
   9 <sinzui> me
  10 <henninge> me/2
  11 <herb> me
  12 * intellectronica (n=tom@intellectronica.net) has joined #launchpad-meeting
  13 <matsubara> only half of you henninge ? :-)
  14 <matsubara> stub, hi
  15 <matsubara> rockstar, hi
  16 <intellectronica> me
  17 <matsubara> bigjools, hello
  18 <rockstar> ni!
  19 <bigjools> me
  20 <henninge> matsubara: yes, sorry, concurrent meeting in #ubuntu-meeting
  21 <matsubara> ok, everyone is here
  22 <matsubara> Ursinha sends apologies as she's still ill
  23 <matsubara> [TOPIC] Agenda 
  24 <MootBot> New Topic:  Agenda
  25 <matsubara>  * Actions from last meeting
  26 <matsubara>  * Oops report & Critical Bugs 
  27 <matsubara>  * Operations report (mthaddon/herb/spm)
  28 <matsubara>  * DBA report (stub)
  29 <matsubara> [TOPIC] * Actions from last meeting
  30 <MootBot> New Topic:  * Actions from last meeting
  31 <matsubara>  * Ursinha to talk to flacoste about buildbot and storm updating for testing when he's available today
  32 <matsubara> I've asked Ursinha and she did that
  33 <matsubara> so, moving on
  34 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  35 <MootBot> New Topic:  * Oops report & Critical Bugs
  36 <matsubara> I have 1 bug and 2 oopses for today. bug 394560 for malone, OOPS-1274A1231 for translations and OOPS-1274ED146 for registry
  37 <ubottu> Launchpad bug 394560 in launchpad "LookupError: unknown encoding: processing email" [Undecided,New] https://launchpad.net/bugs/394560
  38 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1274A1231
  39 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1274ED146
  40 <matsubara> who should I talk to to update ubottu oops urls?
  41 <matsubara> [action] matsubara to chase owner of ubottu to update the oops url
  42 <MootBot> ACTION received:  matsubara to chase owner of ubottu to update the oops url
  43 <matsubara> intellectronica, I need to try to reproduce bug 394560 and add more info
  44 <ubottu> Launchpad bug 394560 in launchpad "LookupError: unknown encoding: processing email" [Undecided,New] https://launchpad.net/bugs/394560
  45 <matsubara> intellectronica, once I do that, do you think you can assign someone for a fix?
  46 <rockstar> matsubara, you should chase the owner with a bit of wood.
  47 * intellectronica looking
  48 <matsubara> rockstar, you mean like a club?
  49 <flacoste> i hate to say this, but this looks like a Foundations issue :-(
  50 <matsubara> hehe
  51 * salgado has quit (Read error: 60 (Operation timed out))
  52 <intellectronica> matsubara: sure, i'll assign to myself and investigate. i have no idea, off the top of my head, what the problem is
  53 <matsubara> flacoste, intellectronica: ok, I'll try to reproduce the issue today and let you know. I'll move the bug to the right project accordingly
  54 <rockstar> matsubara, :)
  55 <flacoste> intellectronica: well, it seems that we are trying to decode the string
  56 <flacoste> based on the encoding specified in the email
  57 <flacoste> and that encoding is x-unknown
  58 <flacoste> which of course doesn't exist
  59 <matsubara> [action] matsubara to reproduce bug 394560, add more info and work with team responsible to have it prioritized
  60 <flacoste> so i'd say from the face of it it's a client-side problem
  61 <MootBot> ACTION received:  matsubara to reproduce bug 394560, add more info and work with team responsible to have it prioritized
  62 <ubottu> Launchpad bug 394560 in launchpad "LookupError: unknown encoding: processing email" [Undecided,New] https://launchpad.net/bugs/394560
  63 <henninge> matsubara: that oops is covered by bug 394224
  64 <ubottu> Launchpad bug 394224 in rosetta "More unique constraints in updateTranslation" [High,Triaged] https://launchpad.net/bugs/394224
  65 <matsubara> henninge, I see it's already triaged and prioritized, so thanks!
  66 * salgado (n=salgado@canonical/launchpad/salgado) has joined #launchpad-meeting
  67 <intellectronica> flacoste: yeah, if that's the case then it's not a bug
  68 <matsubara> sinzui, the other one is a OOPS-1274ED146 a timeout on team membership view
  69 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1274ED146
  70 <matsubara> sinzui, it's issuing > 3000 queries
  71 <matsubara> so it probably can be optimized further
  72 <matsubara> [action] matsubara to file a bug for OOPS-1274ED146
  73 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1274ED146
  74 <MootBot> ACTION received:  matsubara to file a bug for OOPS-1274ED146
  75 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1274ED146
  76 <intellectronica> flacoste: or maybe it's worth coercing x-unknown to None, if that's a common behaviour for mailers
  77 <sinzui> This looks just like the timout for merging teams
  78 <sinzui> matsubara: I will ask Edwin to look into this since the problem look identical
  79 <matsubara> sinzui, thanks. once I have a bug for it I'll assign to him
  80 <matsubara> sinzui, looking at the page where the oops happened, doesn't seem related to team merge
  81 <sinzui> The two repeat queries are the same that I looked at two hours
  82 <matsubara> I see. I'll file a bug anyway and can dupe later on to the team merge bug if that's the case
  83 <matsubara> sinzui, thanks
  84 <sinzui> matsubara: We have a systemic problem in that we do not have a standard method to get a lot of IPerson
  85 <matsubara> sinzui, so the fix for the team merge one will include such a method?
  86 <sinzui> No not at all, We need a big spec to fix a systemic problem
  87 * sinzui created a spec yesterday to end the nightmare of deactivated account being members or owners of other Launchpad objects
  88 <matsubara> sinzui, by spec you mean a blueprint or a new story or something else?
  89 <sinzui> I mean both. It is a project that requires lots of planning and breakdown into work
  90 <matsubara> for the critical bugs: we have two, one in progress which the fix is ready (according to the bug report) and another one in Triaged state
  91 <matsubara> bug 391903 and bug 390563
  92 <ubottu> Launchpad bug 391903 in launchpad-code "Scanner user needs more database permissions" [Critical,In progress] https://launchpad.net/bugs/391903
  93 <ubottu> Launchpad bug 390563 in bzr "absent factory exception from smart server when streaming 2a stacked branches" [Critical,In progress] https://launchpad.net/bugs/390563
  94 <matsubara> rockstar, do you know about the latter one? the lp-code bugtask is in triaged state
  95 <rockstar> matsubara, so, there were two issues here, one client side, one server side.  The server side one has been fixed on Launchpad.
  96 <matsubara> sinzui, all right. where are you keeping track of this work? I'd like to link bugs that might benefit from it to the spec
  97 <stub> bug 391903 should probably not be critical - a patch and test update needs to land this cycle sometime. Production is fine at the moment.
  98 <ubottu> Launchpad bug 391903 in launchpad-code "Scanner user needs more database permissions" [Critical,In progress] https://launchpad.net/bugs/391903
  99 <sinzui> A blueprint. I have not scheduled it since my team may state it requires an infinite number of monkeys to complete
 100 <matsubara> stub, is it not critical anymore because the permissions were granted on the production db?
 101 <stub> I think so. The patch does have to land before next rollout though or that will revert. Does that count as critical?
 102 <matsubara> sinzui, right. let me know the URL please
 103 <flacoste> stub, matsubara: that should simply be high
 104 <matsubara> I'll lower the importance. thanks flacoste and stub 
 105 <matsubara> rockstar, since it was fixed on LP, could you set the bug as fix released?
 106 <sinzui> matsubara: systemic problems are not fixed in critical issues. I am mearly stating that I expect to continue to allocate staff to fix these timeout issues until we have new tools to get masses of users and teams from the db
 107 <matsubara> s/bug/bugtask/
 108 <matsubara> sinzui, the oops I brought to you today is not critical.
 109 <rockstar> matsubara, I think there are some other things we're trying to track (like the client side issue), but I'll raise it in the standup today.
 110 <sinzui> matsubara: So I do not need to fix this this cycle?
 111 <matsubara> sinzui, what I'm trying to do is link those bugs to the blueprint, so when the systemic problem is fixed, we can fix the oopses with the new infrastructure (or at least consider those bugs while implementing the new infrastructure)
 112 <matsubara> sinzui, I don't think they are critical, since it's a soft time out. I'm bringing the issue to you to keep it on your radar
 113 <sinzui> matsubara: I must create a blueprint for this class of problem first
 114 <matsubara> [action] sinzui to create a blueprint about the systemic problem in retrieving lots of IPerson and let matsubara know
 115 <MootBot> ACTION received:  sinzui to create a blueprint about the systemic problem in retrieving lots of IPerson and let matsubara know
 116 <matsubara> rockstar, thanks!
 117 <matsubara> I think that's all for the oops & critical bugs section
 118 <matsubara> thanks everyone
 119 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
 120 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 121 <matsubara> herb, ?
 122 <herb> Since I totally slacked off last week on the OSA report, I'll cover the last two weeks here.
 123 <herb> 2009-06-24 - Production rollout to r2.2.6. Rollout took longer and caused unforseen downtime. Details can be found in the incident report.
 124 <herb> 2009-06-26 - Cherry picked r8204 to codehost
 125 <herb> 2009-06-26 - Cherry picked r8205 to lpnet*, edge* and the scripts server.
 126 <herb> 2009-06-26 - Cherry picked r8701 and r8703 to lpnet* and (part of) soyuz.
 127 <herb> Since the 2.2.6 rollout we've seen a slight uptick in load on the app servers and, early on at least, significant load on the code importers.
 128 <herb> Staging has seen signifcant memory leaks. The app server has often been 5-10GB resident. This is a pretty bug that will clearly need to be resolved before we can push to production.
 129 <flacoste> herb: it's because of the storm update, stub thinks he landed a fix today
 130 <flacoste> herb: plan is to land this storm update on edge once it clears staging
 131 <herb> flacoste: I figured. I'd like to see it running on staging for a couple of days without leaks before seeing it on edge.
 132 <flacoste> herb: sure
 133 <herb> cool
 134 <matsubara> I'll assign bug 393990 to stub and move to -foundations.
 135 <ubottu> Launchpad bug 393990 in launchpad "staging app server using too much memory" [Undecided,New] https://launchpad.net/bugs/393990
 136 <flacoste> matsubara: that's actually a dupe
 137 <herb> matsubara: that's probably a dupe of 390861
 138 <herb> sorry #390861
 139 <matsubara> bug 390861
 140 <ubottu> Launchpad bug 390861 in launchpad-foundations "Appserver memory issues with Storm 0.14" [High,Triaged] https://launchpad.net/bugs/390861
 141 <herb> dammit!
 142 <herb> :)
 143 <matsubara> ah, ok. so it's all fine
 144 <flacoste> matsubara: bug 390861
 145 <matsubara> well, not fine, but at least it's being tracked :-)
 146 <matsubara> thanks herb and flacoste 
 147 <matsubara> let's move on
 148 <matsubara> [TOPIC] * DBA report (stub)
 149 <MootBot> New Topic:  * DBA report (stub)
 150 <stub> Staging is now hopefully running a non-leaking version of Storm (0.14 + r290 from lp:storm). Its been up for maybe an hour now with no sign of memory bloat, so it is looking promising. if all goes well, next step is to land this on launchpad/devel for testing on edge.
 151 <stub> I repaired about 1300 invalid crosslinks between Person and EmailAddress as best I could. Hopefully these were caused by manual updates or since fixed bugs. The other 2.7 million records are fine. Admins or myself should be able to recover any lost permissions of branches if the reclaim account and merge processes don't suffice. garbo-nightly.py is growing detection of this case (code done, tests to go).
 152 <stub> .
 153 <matsubara> question to stub?
 154 <matsubara> ok, I think that's all for today then.
 155 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 156 <herb> stub: is the permissions reclamation process documented?
 157 <matsubara> #endmeeting
 158 <MootBot> Meeting finished at 10:35.
 159 <matsubara> oops, sorry herb 
 160 <herb> no problem
 161 <intellectronica> thanks matsubara
 162 * intellectronica (n=tom@intellectronica.net) has left #launchpad-meeting
 163 * henninge (n=henning@e177213189.adsl.alicedsl.de) has left #launchpad-meeting ("Ex-Chat")
 164 <stub> herb: No - it will depend what the problem is. Most people affected (assuming they are active users) should be able to recover things using their registered email addresses.
 165 <herb> ok
 166 <stub> herb: It should be like a forgotten password...
 167 * stub crosses fingers
 168 <herb> heh. got it.
 169 <sinzui> matsubara: https://blueprints.edge.launchpad.net/launchpad-registry/+spec/efficient-user-sets
 170 <matsubara> sinzui, thanks