1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 09:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 * salgado is now known as salgado-lunch
   5 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   6 * Notify: salgado is offline (simmons.freenode.net).
   7 <matsubara> [TOPIC] Roll Call 
   8 <MootBot> New Topic:  Roll Call
   9 <herb> me
  10 <jtv> me
  11 <sinzui> me
  12 <Ursinha> me
  13 <rockstar> me
  14 <jtv> I'm standing in for henninge today, since he's on sprint.
  15 <matsubara> thanks jtv 
  16 <danilos> me
  17 <BjornT> me
  18 <matsubara> so, if any of you can't make the meeting next week, please coordinate with another teammate to cover for you and add a notice in the Apologies section in the MeetingAgenda page.
  19 <matsubara> flacoste: ping
  20 <intellectronica> me
  21 <matsubara> bigjools: ping
  22 <bigjools> me
  23 <flacoste> me
  24 * henninge (n=henning@port-213-160-23-156.static.qsc.de) has joined #launchpad-meeting
  25 <matsubara> ok, let's move on, stub can join later
  26 <matsubara> [TOPIC] Agenda 
  27 <MootBot> New Topic:  Agenda
  28 <matsubara>  * Actions from last meeting
  29 <matsubara>  * Oops report & Critical Bugs 
  30 <matsubara>  * Operations report (mthaddon/herb/spm)
  31 <matsubara>  * DBA report (DBA contact)
  32 <matsubara> [TOPIC] * Actions from last meeting
  33 <MootBot> New Topic:  * Actions from last meeting
  34 <matsubara>  * bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one - there was, bug 280925
  35 <matsubara>  * intellectronica to work on bug 279561
  36 <matsubara>  * rockstar to check OOPS-1125CEMAIL1
  37 <matsubara>  * bac to take a look at OOPS-1125A165 - bac filed bug 322792
  38 <matsubara>  * Ursinha to check with kiko if any other rollouts will happen this week
  39 <ubottu> Launchpad bug 280925 in launchpad-registry "Project overview page shows obsolete series" [Low,Fix released] https://launchpad.net/bugs/280925
  40 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
  41 <ubottu> Error: Could not parse data returned by Launchpad: The read operation timed out (https://launchpad.net/bugs/279561/+text)
  42 <ubottu> Launchpad bug 322792 in launchpad-bazaar "Attempting traversal past an unknown object causes an OOPS" [Undecided,Fix released] https://launchpad.net/bugs/322792
  43 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
  44 <Ursinha> holy crap
  45 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  46 <Ursinha> that bug is timing out
  47 <Ursinha> or what?
  48 <Ursinha> anyway
  49 <matsubara> intellectronica: any news about the api bug? 
  50 <Ursinha> my items were done
  51 <matsubara> rockstar: what's up about that oops?
  52 <intellectronica> matsubara: sorry, no news yet
  53 <Ursinha> matsubara, he landed a fix for that
  54 <matsubara> Ursinha: who's he?
  55 <matsubara> :-)
  56 <Ursinha> matsubara, rockstar :)
  57 <Ursinha> sorry
  58 <rockstar> matsubara, I got an RC in for that one.
  59 <matsubara> ok, I remember that one. 
  60 <matsubara> so I guess the only pending one is the api bug which is a mistery to everyone
  61 <matsubara> the good news is that intellectronica found out another thing about that bug that might lead to its root cause
  62 <matsubara> intellectronica: thanks for keeping us posted in the report
  63 <matsubara> let's move on
  64 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  65 <MootBot> New Topic:  * Oops report & Critical Bugs
  66 <matsubara> so in today's oops section I'd like to talk about the timeout bugs you guys are working for the LPW
  67 <matsubara> https://dev.launchpad.net/PerformanceWeeks/February2009
  68 <matsubara> I'm going to review all the landings related to LPW work and add to that page.
  69 <matsubara> so if you wanna help, point me to revision numbers on RF related to that work
  70 <jtv> matsubara: bug 324264 is now Fix Committed.
  71 <ubottu> Launchpad bug 324264 in rosetta "Speed up +translations" [High,Fix committed] https://launchpad.net/bugs/324264
  72 <sinzui> matsubara: r7705 for Bug: 325321
  73 <Ursinha> jtv, great
  74 <matsubara> intellectronica, BjornT : any news about the bug number 1 time out?
  75 <sinzui> matsubara: EdwinGrubbs will land his branch today
  76 <danilos> work on bug 302798 is in progress in different ways (there's a commit from 323something which disabled external suggestions to give us a better idea on how stuff is working, and henning is working on removal of obsolete translations which will reduce our DB size by ~33%)
  77 <ubottu> Launchpad bug 302798 in rosetta "Timeout on +translate page" [High,Triaged] https://launchpad.net/bugs/302798
  78 <BjornT> matsubara: me, intellectronica, and allenap are working on it
  79 <jtv> danilos: hey, I was putting that paragraph together!
  80 <BjornT> matsubara: allenap is looking at reducing the time it take to render the comments, possibly by not showing all by default
  81 <danilos> jtv: ok, I'll let you handle it all from now on :)
  82 <BjornT> matsubara: intellectronica is working on loading the subscribers portlet in a different request
  83 <danilos> matsubara: you can have full trust in jtv as far as PW is concerned :)
  84 <jtv> *cough*
  85 <BjornT> matsubara: and i'm working on optimizing code, based off profiling information
  86 <matsubara> danilos: I do! he's been very helpful with the status updates
  87 <BjornT> matsubara: also, intellectronica's inital tests on dogfood were successful, reducing the time quite a lot :)
  88 <matsubara> great
  89 <flacoste> matsubara: i'm working on bug 316881 (which isn't an OOPS per-se, but related to performance anyhow)
  90 <ubottu> Launchpad bug 316881 in launchpad "Page headers not suitable for HTTP caching" [High,In progress] https://launchpad.net/bugs/316881
  91 <matsubara> stub: there's an email from jono asking for some help with the +project-cloud oops, so if you could help out there, would be awesome
  92 <matsubara> thanks flacoste, i'll add it to the page
  93 <stub> I've replied on the bug. Not sure if I'm helpful though.
  94 <matsubara> stub: cool. thank you
  95 <matsubara> bigjools: news in soyuz. how about the one muharem is taking care of?
  96 <matsubara> s/./?/
  97 <bigjools> matsubara: it's not going so well unfortunately
  98 <bigjools> I don't expect any progress this week
  99 <matsubara> [action] matsubara to add 316881 to foundations section in LPW wiki page
 100 <MootBot> ACTION received:  matsubara to add 316881 to foundations section in LPW wiki page
 101 <matsubara> bigjools: why is that? 
 102 <bigjools> matsubara: the first attempt to fix it failed.  he's also been at the distro sprint this week
 103 <matsubara> bigjools: oh right. well, you guys are excused since the whole team is sprinting and you already landed 2(?) timeout fixes :-)
 104 <bigjools> matsubara: thanks :)
 105 <matsubara> I guess that's it from me. Ursinha, anything else?
 106 <Ursinha> matsubara, no, the pending oops for soyuz I already talked with bigjools
 107 <bigjools> yeah, get edge updating again and we'll see how it went
 108 <matsubara> great. thanks everyone!
 109 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
 110 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 111 <herb> - 2009-01-30 - Friday we updated lpnet, edge and the scripts servers to to r7676.
 112 <herb> - 2009-02-04 - Yesterday we updated codebrowse to r43
 113 <herb> - I feel like I'm starting to sound like a broken record... But we're still being bothered daily, often multiple times, by bug #156453 and bug #118625 which seem to be related.
 114 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
 115 <herb> - We also continue to run into issues associated with bug #260171
 116 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
 117 <ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
 118 <flacoste> herb: i herd mwhudson was working on loggerhead performance this week
 119 <flacoste> as part of LPW
 120 <matsubara> yes!
 121 <sinzui> herb: There is a sprint in the planning to fix that too
 122 <Ursinha> yes, he is
 123 <matsubara> mwhudson is indeed working on that
 124 <flacoste> herb: and well-placed sources also told me that a sprint is being organized to fix those damn issues
 125 <flacoste> herb: so there is hope!
 126 <herb> excellent. a fix would be huge for the LOSAs
 127 <rockstar> herb, mwhudson is on the verge of insanity tracking loggerhead issues down.
 128 <herb> thanks for all the work on that. it is much appreciated.
 129 <matsubara> thanks herb 
 130 <matsubara> [TOPIC] * DBA report (stub)
 131 <MootBot> New Topic:  * DBA report (stub)
 132 <stub> The production dbs seem to be ticking away nicely.
 133 <stub> Staging db updates are being disabled by the losas for some testing, so expect a drop in timeout OOPSES.
 134 <stub> I've had some db patches for this cycle come through already, which is great.
 135 <stub> 3
 136 <stub> 2
 137 <stub> 1
 138 <matsubara> all right. thanks stub 
 139 <flacoste> what about the staging issues?
 140 <matsubara> herb: ^
 141 <flacoste> staging isn't available at the moment
 142 <herb> it's restoring
 143 <flacoste> but we didn't understand why it failed?
 144 <flacoste> or did we?
 145 <matsubara> herb: how long will it take to restore?
 146 <flacoste> and it's just that gmb is done with the testing and we can resume normal staging updates?
 147 <herb> matsubara: should be back up within the next couple of hours.
 148 <Ursinha> herb, about 10 hours ago I was talking with spm and he was unsuccessful to put staging on again
 149 <herb> flacoste: upgrade.py failed because there will still connections open to the staging db.  This left the DB in an indeterminate state, so we had to go through the full restore process with a new copy of the staging DB.
 150 <gmb> flacoste: I'm not done yet.
 151 <gmb> flacoste: I need staging to be up for that
 152 <flacoste> so this means that we are still screwed?
 153 <stub> This is all useful information for read only launchpad btw.
 154 <flacoste> we either need to fix upgrade.py to work without a db restore
 155 <flacoste> or to turn off staging ugprades for the duration of gmb's test
 156 <herb> flacoste: upgrade.py works fine
 157 <herb> and we'll need to turn off upgrades while gmb's testing, per stubs note above.
 158 <flacoste> why were there still connections open?
 159 <stub> upgrade.py cannot work when there are active db connections, as upgrade requires exclusive locks on all the replicated db tables.
 160 <herb> flacoste: the rollout process shuts down the app servers, but there are still potentially cron scripts running, etc.
 161 <flacoste> why usually is this not a problem then?
 162 <stub> Because the restore, replicate and upgrade process is done as a fresh db. Once it is finished, it is swapped into place.
 163 <flacoste> ah right!
 164 * bigjools has to run, will catch scrollback later if you need anything from me
 165 <flacoste> so when I said fix upgrade.py, i should have said 'fix rollout process'
 166 <flacoste> so I guess it's best to simply disbale upgrade for now
 167 <flacoste> i also heard that gmb might do his tests elsewhere
 168 <flacoste> so that might becomes a moot issue
 169 <flacoste> but like stub said, very instructive for read-only launchpad
 170 <gmb> flacoste: Well, that's an embryonic idea right now. I still need a staging / demo machine for the forseeable future.
 171 <matsubara> I want to make an action item for the fix rollout process thing
 172 <matsubara> not sure who would be responsible for that
 173 <matsubara> losas?
 174 <herb> whoa
 175 <flacoste> with the help of stub probably
 176 <matsubara> [action] losas and stub to fix rollout process to avoid the staging restore problems
 177 <MootBot> ACTION received:  losas and stub to fix rollout process to avoid the staging restore problems
 178 <herb> there isn't anything inherently wrong with the rollout process.
 179 <herb> but ok
 180 <matsubara> thanks everyone
 181 <matsubara> anythign else before I close?
 182 <flacoste> well, it's currently not reliable if we don't do a DB restore it seems
 183 <flacoste> nope
 184 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 185 <matsubara> #endmeeting
 186 <MootBot> Meeting finished at 09:36.