1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 09:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 * flacoste (n=francis@h-213.61.60.234.host.de.colt.net) has joined #launchpad-meeting
   6 <matsubara> [TOPIC] Roll Call 
   7 <MootBot> New Topic:  Roll Call
   8 <bigjools> me
   9 <Ursinha> me
  10 <sinzui> me
  11 <henninge> ich
  12 <herb> me
  13 <flacoste> me
  14 <gmb> me
  15 <matsubara> stub is missing.
  16 <matsubara> well, let's continue. stub can join later
  17 <gmb> Presumed hairy.
  18 <matsubara> [TOPIC] Agenda 
  19 <MootBot> New Topic:  Agenda
  20 <matsubara>  * Actions from last meeting
  21 <matsubara>  * Oops report & Critical Bugs 
  22 <matsubara>  * Operations report (mthaddon/herb/spm)
  23 <matsubara>  * DBA report (DBA contact)
  24 <matsubara>  * Next Launchpad Performance Week
  25 <Ursinha> traditionally he'll join in a few moments
  26 <matsubara> :-)
  27 <matsubara> [TOPIC] * Actions from last meeting
  28 <MootBot> New Topic:  * Actions from last meeting
  29 <matsubara> * stub to investigate the fix to avoid staging restore problems
  30 <matsubara>  * rockstar to fix https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1138CEMAIL12
  31 <matsubara>  * sinzui to have someone to fix bug 237722
  32 <matsubara>  * Ursinha to file bugs to https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1138EC328 - bug 329917, and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1138B2582 - bug 329908
  33 <matsubara>  * intellectronica to investigate https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1137F2893
  34 <ubottu> Launchpad bug 237722 in launchpad-registry "Cyclical team membership allowed in +editproposedmembers" [Low,Triaged] https://launchpad.net/bugs/237722
  35 <ubottu> Launchpad bug 329917 in malone "Changing a task's target using the API OOPSes as NotImplementedError" [Undecided,New] https://launchpad.net/bugs/329917
  36 <ubottu> Launchpad bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New] https://launchpad.net/bugs/329908
  37 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1138B2582
  38 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1137F2893
  39 <Ursinha> mine is fine, filed bugs are there
  40 <sinzui> matsubara: Not done, but it was discussed
  41 <matsubara> sinzui, thanks. I see that's triaged as low priority so it's ok
  42 * sinzui will pick a victim to close the bug
  43 <Ursinha> :)
  44 <sinzui> matsubara: the person who discovers the problem has the power to fix it. He is often the person who put the teams in the situation
  45 <sinzui> matsubara: so we believe the right answer is to inform the user what he is doing before he gets the warning he has done something impossible
  46 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  47 <Ursinha> :)
  48 <matsubara> sinzui, inform the user or just block the user from doing it?
  49 <matsubara> hi stub
  50 <matsubara>  * stub to investigate the fix to avoid staging restore problems
  51 <matsubara> any news about this?
  52 <matsubara> [action] matsubara to chase rockstar about a fix for OOPS-1138CEMAIL12
  53 <MootBot> ACTION received:  matsubara to chase rockstar about a fix for OOPS-1138CEMAIL12
  54 <stub> No news
  55 <sinzui> matsubara: both actually. Explain that launchpad has removed a pending membership because between x in y. If this is wrong the user should remove the member ship he just approved and switch propose the team again
  56 <matsubara> sinzui, right
  57 <matsubara> gmb, any news about OOPS-1137F2893?
  58 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1137F2893
  59 <gmb> matsubara: Not as far as I know; will confirm with intellectronica.
  60 <Ursinha> matsubara, iirc flacoste was discussing that with intellectronica
  61 <gmb> intellectronica's sprinting atm so let's move on and I'll get back to you when I know more.
  62 <matsubara> Ursinha, gmb: ok thanks
  63 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  64 <MootBot> New Topic:  * Oops report & Critical Bugs
  65 <Ursinha> all right
  66 <Ursinha> one bug for bugs, two OOPSes for soyuz, one OOPS for foundations
  67 <Ursinha> for bugs: bug 329908
  68 <ubottu> Launchpad bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New] https://launchpad.net/bugs/329908
  69 <bigjools> go soyuz
  70 <flacoste> matsubara: yes, we discussed a solution, i don't know the status of the fix though
  71 <Ursinha> gmb, that one is happening quite frequently
  72 <Ursinha> on a daily basis, but only a few
  73 <matsubara> flacoste, okie. thanks
  74 <gmb> Ursinha: Hmm. Okay.
  75 <gmb> Ursinha: That sounds - at first glance - like a replication problem again.
  76 <gmb> But that's just a stab in the dark
  77 <Ursinha> gmb, I didn't stick to this idea because it's happening like every day
  78 <Ursinha> do you think it's possible to be a replication problem even considering this?
  79 <gmb> Ursinha: Well, last time something like that happened it was  a replication problem, but that was fixed last cycle.
  80 <gmb> Ursinha: Frankly, at this point, I've no idea. I'll look into it.
  81 <Ursinha> gmb, I remember that
  82 <Ursinha> thanks gmb
  83 * Ursinha looks at bigjools 
  84 <Ursinha> bigjools,  https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
  85 <bigjools> ok
  86 <Ursinha> what do you say?
  87 <bigjools> first one was caused by cprov
  88 <bigjools> I will check with him
  89 <cprov> I did it on purpose
  90 <bigjools> is there a bug filed?
  91 <stub> Librarian OOPSes are not caused by replication - the Librarian only talks to the master database.
  92 <Ursinha> bigjools, I didn't file one, but can do that
  93 <bigjools> second one looks nasty, we'll look into it
  94 <Ursinha> gmb, ^
  95 <cprov> bigjools: the second oops need investigation
  96 <gmb> stub, Ursinha: Curse. Oh well, I'll have to do some work then.
  97 <bigjools> Ursinha: can you file bugs on those for me please
  98 <Ursinha> matsubara, [action] cprov and bigjools to investigate OOPS-1145EA14
  99 <Ursinha> bigjools, sure
 100 <bigjools> thanks muchly
 101 <Ursinha> matsubara, [action] Ursinha to file bugs for https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
 102 <matsubara> [action] cprov and bigjools to investigate OOPS-1145EA14
 103 <MootBot> ACTION received:  cprov and bigjools to investigate OOPS-1145EA14
 104 <matsubara>  [action] Ursinha to file bugs for https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1143EB189 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1145EA14
 105 <Ursinha> bigjools, np :)
 106 <Ursinha> the last one for foundations:  https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1146XMLP1
 107 <Ursinha> flacoste, ^
 108 <stub> gmb: We have seen similar 'impossible' situations before, and have failed to work it out. No requests to that URL should be made until the transaction that created it commits, and as soon as it commits not founds shouldn't happen.
 109 <gmb> stub: Hmm. Okay. Thanks.
 110 <flacoste> Ursinha: that's not a bug
 111 * gmb wonders about the possibility of getting a `sleep(10)` into apport...
 112 <flacoste> Ursinha: that's something posting non-XMLRPC request to the xmlrpc server
 113 <flacoste> Ursinha: so find what is doing that and stop it from doing it :-)
 114 <flacoste> Ursinha: probably a Mailman bad request for some reason (restart?)
 115 <stub> gmb: It might help. I suspect commit is returning before the commit really has finished to make things look fast (suspect because everything else seems impossible)
 116 <Ursinha> flacoste, hm, right
 117 <Ursinha> flacoste, I'll keep watching if we have more of those
 118 <gmb> stub: This is Launchpad; we eat impossible things for breakfast.
 119 <flacoste> yeah, if it happens often there is probably either a bug in Mailman or a deployment issue
 120 <Ursinha> flacoste, fair enough
 121 <Ursinha> that's fine
 122 <Ursinha> all for me
 123 <matsubara> gmb, that could be the new slogan :-)
 124 <Ursinha> lol
 125 <matsubara> thanks Ursinha 
 126 <Ursinha> thanks everyone
 127 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
 128 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 129 <herb> It's been a quiet week. No cherry picks, no major service affecting problems.
 130 <herb> Just to make sure it's staying on everyone's radar, we continue to have daily problems related to bug #156453 and bug #118625.
 131 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
 132 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
 133 <herb> Further we contine to have problems every couple of days related to bug #260171.
 134 <ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
 135 <herb> We need to come up with a plan on how to handle Bug #327423 and bug #327455. Just throwing more RAM at the problem isn't likely to be a workable solution. We're happy to do anything we can do to help reduce the impact of these bugs, but we clearly need some input from the devs.
 136 <ubottu> Launchpad bug 327423 in launchpad-registry "Memory usage of the karma update script foaf-update-karma-cache.py seems excessive" [Low,Triaged] https://launchpad.net/bugs/327423
 137 <ubottu> Launchpad bug 327455 in rosetta "language-pack-exporter.py and memory usage" [Undecided,New] https://launchpad.net/bugs/327455
 138 <herb> Thats all from us, unless there are questions.
 139 <matsubara> sinzui, can you raise the importance of #327423?
 140 <sinzui> I can
 141 <matsubara> henninge, can you comment and triage accordingly #327455?
 142 <henninge> I know that danilo and jtv worked on that and are now able to produce language packs again.
 143 <henninge> I don't know how they did that, though.. ;)
 144 <herb> henninge: I think they had to use the DB server to make it work though.
 145 <henninge> I will poke danilo to commtnt
 146 <matsubara> thanks henninge 
 147 <henninge> herb: ok, so it is not really solved, then?
 148 <herb> henninge: to the best of my knowledge, no.
 149 <henninge> our team communication is not at its best this week as 2/3 are sprinting heavily ... :-/
 150 <herb> henninge: or at least not in a permanent way.
 151 <matsubara> all right. thanks herb 
 152 <herb> thanks
 153 <matsubara> [TOPIC] * DBA report (DBA contact)
 154 <MootBot> New Topic:  * DBA report (DBA contact)
 155 <stub> The production systems are now using the standard startup/shutdown script to control the slony daemons. The wiki documentation has been updated. LOSAs should have sudo access to run the /etc/init.d script.
 156 <stub> I'll be investigating database connection load balancers to make best use of the new hardware. pg_pool, pg_pool 2 and pgbouncer are all options I'm aware of to distribute slave connections evenly across the slave backends.
 157 <stub> fini.
 158 <matsubara> questions for stub?
 159 <matsubara> ok, let's move on
 160 <matsubara> [TOPIC] * Next Launchpad Performance Week
 161 <MootBot> New Topic:  * Next Launchpad Performance Week
 162 <matsubara> I updated https://dev.launchpad.net/PerformanceWeeks/April2009
 163 <matsubara> with a few page ids that we could tackle for the next performance week
 164 <matsubara> if you have more page ids/oopses that you want to work on in the next LPW, please add to the list
 165 <matsubara> that's all from me
 166 <matsubara> anything else before I close?
 167 <Ursinha> nope
 168 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
 169 <matsubara> #endmeeting 
 170 <MootBot> Meeting finished at 09:31.