1 <Ursinha> #startmeeting
   2 <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   3 <Ursinha> [TOPIC] Roll Call
   4 <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
   5 <Ursinha> me
   6 <MootBot> Meeting started at 10:00. The chair is Ursinha.
   7 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   8 <MootBot> New Topic:  Roll Call
   9 <bigjools> me
  10 <sinzui> me
  11 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  12 <herb> me
  13 <danilos> me
  14 <stub> me
  15 <Ursinha> intellectronica, rockstar, hi
  16 <intellectronica> me
  17 <rockstar> me
  18 <Ursinha> hey!
  19 <Ursinha> so we're all here
  20 <Ursinha> apologies from matsubara, he's not here today
  21 <Ursinha> [TOPIC] Agenda
  22 <Ursinha>  * Actions from last meeting
  23 <Ursinha>  * Oops report & Critical Bugs
  24 <Ursinha>  * Operations report (mthaddon/herb/spm)
  25 <Ursinha>  * DBA report (stub)
  26 <MootBot> New Topic:  Agenda
  27 <Ursinha> [TOPIC] * Actions from last meeting
  28 <Ursinha> * matsubara to check with Ursula about last week's meeting action itens - Items were done
  29 <Ursinha> * flacoste to approve CP requests
  30 <MootBot> New Topic:  * Actions from last meeting
  31 <Ursinha> my items were okay
  32 <stub> and apologies from Francis
  33 <Ursinha> hmm, I missed flacoste
  34 <Ursinha> stub, so he's not coming today..
  35 <Ursinha> I'll have to check with him later then
  36 <stub> I'm designated victim
  37 <Ursinha> oh :)
  38 <Ursinha> stub, so, do you know if he did that?
  39 <stub> Nope :)
  40 * afk is now known as mthaddon
  41 <danilos> there were a bunch of CPs done afaik
  42 <herb> Ursinha: he did. they've been rolled out.
  43 <Ursinha> herb, awesome
  44 <stub> No outstanding CP requests
  45 <Ursinha> danilos, I'm seeing them in lp list
  46 <Ursinha> allright
  47 <Ursinha> thanks guys
  48 <Ursinha> moving on
  49 <Ursinha> [TOPIC] * Oops report & Critical Bugs
  50 <Ursinha> the bugs I had I already discussed with people 
  51 <Ursinha> I don't think I need to mention them here again :)
  52 <Ursinha> all critical bugs are Fix Committed, so we're okay
  53 <Ursinha> unless someone have something to say here, let's move on
  54 <MootBot> New Topic:  * Oops report & Critical Bugs
  55 <danilos> well, there's one that's not
  56 <Ursinha> danilos, which one?
  57 <danilos> pytz installation, let me look up the bug number
  58 <danilos> bug 388825
  59 <ubottu> Launchpad bug 388825 in rosetta "UnknownTimeZoneError on various pages" [Critical,Triaged] https://launchpad.net/bugs/388825
  60 <sinzui> That will fix a bug in the registry
  61 <danilos> Ursinha: we need help from LOSAs to resolve that, but there's more to it than that
  62 <danilos> i.e. how do we prevent this from happening
  63 * flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
  64 <Ursinha> danilos, let me look, I checked half an hour ago and it wasn't there :)
  65 <Ursinha> danilos, hm, I see
  66 <danilos> Ursinha: stub marked it as critical 4 hours ago :)
  67 <sinzui> I think bug 155334 will be fixed with the pytz landing
  68 <ubottu> Launchpad bug 155334 in pytz "In select timezone section, "Kolkata" is shown as "Calcutta"" [Undecided,Fix released] https://launchpad.net/bugs/155334
  69 <Ursinha> danilos, so refresh fail :)
  70 <Ursinha> Undecided and Fix released
  71 <Ursinha> is that correct?
  72 <stub> pytz is updated on edge, so production is now seeing timezones it can't cope with. Fail.
  73 <Ursinha> stub, argh, I understand the problem
  74 <Ursinha> danilos, so you need a losa to help you with that
  75 * sinzui moves Kolkata for a moment
  76 <danilos> Ursinha: well, I need losa to try to fix it, we are totally not in power of doing anything about it
  77 <Ursinha> danilos, okay
  78 <danilos> Ursinha: this is about date time formatter in TAL which breaks
  79 <stub> I thought I had assigned it to Tom already?
  80 <herb> stub, danilos: we can discuss it after the meeting.
  81 <Ursinha> herb, can you help with that?
  82 <Ursinha> oh, awesome
  83 <Ursinha> :)
  84 <danilos> herb: sure, thanks
  85 <Ursinha> thanks herb
  86 <sinzui> stub: danilos Who do I give credit to for fixing pytz?
  87 <Ursinha> [action] danilos to discuss with herb a fix for bug 388825
  88 <danilos> stub: can you join us in that discussion as well? (you seem to know more about it than I do :)
  89 <MootBot> ACTION received:  danilos to discuss with herb a fix for bug 388825
  90 <ubottu> Launchpad bug 388825 in rosetta "UnknownTimeZoneError on various pages" [Critical,Triaged] https://launchpad.net/bugs/388825
  91 <herb> danilos, stub: I think we can keep it pretty short so stub can get offline for a few hours.
  92 <danilos> herb: sure
  93 <Ursinha> okay
  94 <Ursinha> anything else here?
  95 <Ursinha> moving on then
  96 <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
  97 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
  98 <herb> 2009-06-12: Cherry picked r8488, r8508 and r8527 to most (all?) of soyuz.
  99 <herb> 2009-06-16: Cherry picked r8608 to lpnet* and the librarian.
 100 <herb> 2009-06-17: Cherry picked r8644, r8647 and bzr 1.16rc1 to just about everything. I think we only skipped soyuz.
 101 <herb> Since the CP yesterday we've had to restart codebrowse a couple of times. Apparently it's taking a while to restart (10 minutes) because it's trying to fetch wadllib and timing out before starting. I don't have much more detail than that unfortunately.
 102 <Ursinha> rockstar, are you aware of that?
 103 <rockstar> Ursinha, I indeed was, although not involved, so cannot provide support.
 104 <Ursinha> rockstar, okay, can you ask who could help then? :)
 105 <Ursinha> herb, hmm, I guess we don't have a bug for that, do we?
 106 <herb> Ursinha: I don't know. I saw it in scrollback when I started this morning.  I haven't experienced the problem and haven't had a chance to check with spm or mthaddon to figure out if a bug was filed.
 107 <rockstar> Ursinha, I think everyone knows what's going on.  spm, jml, and mwhudson are quite the think tank.
 108 <rockstar> Everyone == "the people who really can do something about it"
 109 <Ursinha> rockstar, ok, I'll talk to matsubara and make him aware of that, as he'll be doing the late shift today
 110 <Ursinha> thanks rockstar and herb
 111 <mthaddon> yeah, I'm not aware of a bug report - happened after my EOD so I heard a little about it but not the full details
 112 <Ursinha> mthaddon, I'll catch up with spm later, thanks!
 113 <Ursinha> anything else for herb
 114 <Ursinha> ?
 115 <Ursinha> [action] Ursinha to talk to matsubara and spm later about the codebrowse taking 10 mins to restart
 116 <MootBot> ACTION received:  Ursinha to talk to matsubara and spm later about the codebrowse taking 10 mins to restart
 117 <Ursinha> moving on then
 118 <Ursinha> [TOPIC] * DBA report (stub)
 119 <MootBot> New Topic:  * DBA report (stub)
 120 <danilos> yeah, entire LOSA team was a great help for QA last week, my thanks :)
 121 <stub> Simulated replication lag has been turned off on staging as it was triggering the lag detection we use to make batch jobs play nicely, blocking them from running. Replication settings on staging now match production on the Launchpad trunk. I can reset things manually again if further testing is needed and there is difficulty updating the staging code base at the moment.
 122 <stub> The Storm update has landed, and been reverted. There are issues with buildout and buildbot. Perhaps buildout is running a script without PYTHONPATH set correctly, perhaps it is a catch-22 where the environment setup process invokes scripts that require Storm which isn't found because the environment isn't setup yet. Bug #388825 reassigned to Francis as I think he is the only one around at the moment with access to investigate further?
 123 <stub> autovacuum settings have been made less agressive on production and staging. Its still more agressive than the default. I'll keep an eyeball on it to see if it needs tweaking up or down further.
 124 <ubottu> Launchpad bug 388825 in rosetta "UnknownTimeZoneError on various pages" [Critical,Triaged] https://launchpad.net/bugs/388825
 125 <stub> The autovacuum change was prompted by seeing hung autovacuum jobs on staging (again) and reports of testrun failures from possibly hung vacuum processes (again). Vacuum is much nicer under PG 8.4, so I suspect we will live with it or work around it for a few months as getting anything useful for upstream will be a pain.
 126 <stub> Erm... wrong bug.
 127 <Ursinha> :)
 128 <stub> Bug #352965
 129 <ubottu> Launchpad bug 352965 in launchpad-foundations "Update Storm" [Medium,In progress] https://launchpad.net/bugs/352965
 130 <stub> PG 8.4 RC1 is out now, yay.
 131 <danilos> stub: are we doing any testing with PG 8.4? is it worth starting it?
 132 <stub> Not much point until release - we only do pretty standard stuff, so unless we run it on production we are unlikely to find anything other will not find too.
 133 * herb suspects it's unlikely we'll move to 8.4 until the next LTS, unless it's going to be backported to hardy.
 134 <herb> of course I can be outvoted on that.
 135 <stub> It needs to be backported to hardy or 8.3 supported under LTS anyway or upgrades will be difficult.
 136 <stub> Anyway - no need to make that call yet.
 137 <danilos> yeah, let's move on; Ursinha? :)
 138 <Ursinha> danilos, I'm seeing people discuss about that
 139 <Ursinha> because it's important, but as stub said, no need to make that call yet
 140 <Ursinha> moving on then, per danilos request :)
 141 <danilos> perhaps another question first :)
 142 <Ursinha> next topic is the end of this meeting
 143 <danilos> stub: what's going to happen with storm upgrade? post 2.2.6 or will you try to figure it out?
 144 <stub> It is out of my hands - needs people with buildbot access to investigate further.
 145 <danilos> stub: ok, thanks
 146 <stub> The branch is ready and waiting to land (again)
 147 <Ursinha> stub, who could help you with that?
 148 <danilos> stub: perhaps we can test it on staging today even if it's not landed?
 149 <Ursinha> testing, that is
 150 <danilos> stub: I'd hate to have a change that big land (I am referring to storm upgrade itself) without proper QA for 2.2.6
 151 <Ursinha> +1
 152 <stub> Ursinha: I've assigned the bug to Francis. As far as I'm aware, only Francis and Gary have the required buildbot access to look further. If other people also have access and the debugging skills, they can look at it too.
 153 <stub> There is no reason we can't test it on staging if we want.
 154 <Ursinha> stub, good.
 155 <stub> Someone can make another attempt at switching from buildout back to keeping storm in sourcecode if they want - I just tried and failed, but it is new to me too.
 156 <Ursinha> stub, I see, I'll talk to flacoste about that later today
 157 <Ursinha> anything else here?
 158 <Ursinha> [action] talk to flacoste about buildbot and storm updating for testing when he's available today
 159 <MootBot> ACTION received:  talk to flacoste about buildbot and storm updating for testing when he's available today
 160 <Ursinha> okay
 161 <Ursinha> let's finish this
 162 <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
 163 <Ursinha> #endmeeting