1 <MootBot> Meeting started at 09:00. The chair is matsubara_.
   2 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   3 <matsubara_> so, welcome to this week's production meeting
   4 <matsubara_> who's here today?
   5 <mrevell> me
   6 <rockstar> me
   7 <sinzui> me
   8 <intellectronica> me
   9 <danilos> me
  10 <flacoste> me
  11 <bigjools> me
  12 <Ursinha> me
  13 <matsubara_> herb: ?
  14 <herb> me
  15 <matsubara_> ok everyone here but stub. I've pinged him already. let's move on and stub can catch up later
  16 <matsubara_> [TOPIC] Agenda
  17 <matsubara_>  * Next meeting
  18 <matsubara_>  * Actions from last meeting
  19 <matsubara_>  * Oops report & Critical Bugs 
  20 <matsubara_>  * Operations report (mthaddon/herb/spm)
  21 <matsubara_>  * DBA report (DBA contact)
  22 <MootBot> New Topic:  Agenda
  23 <matsubara_> [topic] next meeting
  24 <MootBot> New Topic:  next meeting
  25 <matsubara_> so, same time next week?
  26 <intellectronica> i won't be here next week. i'm on holiday
  27 <matsubara_> intellectronica: can you arrange with someone from your team to send a qa contact?
  28 <intellectronica> matsubara_: sure
  29 <danilos> same time next week is just fine
  30 <matsubara_> [action] add intellectronica to apologies on MeetingAgenda.
  31 <MootBot> ACTION received:  add intellectronica to apologies on MeetingAgenda.
  32 <matsubara_> all right then, same time next week.
  33 <matsubara_> [topic] actions from last meeting
  34 <MootBot> New Topic:  actions from last meeting
  35 <matsubara_> no actions from last meeting.
  36 <matsubara_> there were a couple of old ones which i cleaned up
  37 <matsubara_> [topic] oops report & critical bugs
  38 <MootBot> New Topic:  oops report & critical bugs
  39 <Ursinha> okay
  40 <matsubara_> Ursinha: please take the stage
  41 <Ursinha> it will be fast
  42 <Ursinha> have one oops, want to know if someone of bugs (which I presume to be the owner) knows about that
  43 <Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1047S138
  44 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1047S138
  45 * intellectronica looking
  46 <Ursinha> intellectronica, thanks
  47 <matsubara_> hmm that's probably abel's turf
  48 <Ursinha> matsubara_, yes
  49 <intellectronica> i'll relay that to abel. it's black magic to me, tbh
  50 <Ursinha> intellectronica, ok, thanks
  51 <Ursinha> may I file a bug about it?
  52 <Ursinha> intellectronica, I'll do that and give you the #
  53 <matsubara_> should have already :-)
  54 <Ursinha> anyway
  55 <intellectronica> Ursinha: plesae do
  56 <Ursinha> matsubara_, just saw it man :)
  57 <Ursinha> all other bugs are being handled, so no need to mention
  58 <matsubara_> Ursinha: fair enough
  59 <Ursinha> re critical bugs
  60 <Ursinha> we have one, triaged
  61 <Ursinha> code people
  62 <matsubara_> bug 297448
  63 <ubottu> Launchpad bug 297448 in launchpad-bazaar "Launchpad will not do initial mirror of a branch that's the development focus of a project" [Critical,Triaged] https://launchpad.net/bugs/297448
  64 <Ursinha> bug 297448
  65 <matsubara_> rockstar: any news about that one?
  66 <rockstar> matsubara_, there has been discussion about it, but I'm not sure it's being worked.  That makes me question the Critical status.
  67 <matsubara_> exactly, can you down the importance to high or find someone to work on it? critical bugs need to be worked on immediately and a IncidentLog reported
  68 <rockstar> I think me all know the cause.  I'll bring it up in the meeting tonight, and get it assigned and fixed or downgrade the status
  69 <rockstar> :)
  70 <matsubara_> great. thanks rockstar!
  71 <rockstar> I think we're on the same page about this.  :)
  72 <bigjools> bingo
  73 <Ursinha> great
  74 <matsubara_> great. I think that's all for the oops & critical bugs section
  75 <matsubara_> thanks everyone
  76 <Ursinha> thanks guys
  77 <matsubara_> [topic] * Operations report (mthaddon/herb/spm)
  78 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
  79 <herb> - 2008-11-10: Cherry pick r7246 the app servers and private xmlrpc server to fix bug #288064
  80 <ubottu> Launchpad bug 288064 in launchpad-foundations "Some timeout oops reports seem incomplete" [High,Fix committed] https://launchpad.net/bugs/288064
  81 <herb> - 2008-11-12: Cherry pick r7257 to the scripts server to fix bug #286362
  82 <ubottu> Launchpad bug 286362 in rosetta "Import failure: 'POFileImporter' object has no attribute 'addUpdateError'" [High,Fix released] https://launchpad.net/bugs/286362
  83 <herb> - Bugs #118625 and #156453 continue to be a problem. We again had to restart codebrowse several times this week.
  84 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
  85 <herb> - Bug #239569 became a real problem this week when coupled with some sort of svn issue with the codeimport worker, as was noted in my email to the list. It ended up effectively DoSing the DB. We've put limits in at the DB level to hopefully prevent this in the future, but we should raise the importances of this bug.
  86 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
  87 <ubottu> Bug 239569 on http://launchpad.net/bugs/239569 is private
  88 <herb> - We have a production rollout coming up on 2008-11-19. Have we nailed down the time on that yet?
  89 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  90 <danilos> hey stub :)
  91 <danilos> herb: I suppose time is something you guys decide on with kiko, no?
  92 <matsubara_> hey Ursinha, since the cp for bug 288064, did you notice if the timeouts are now sane again?
  93 <ubottu> Launchpad bug 288064 in launchpad-foundations "Some timeout oops reports seem incomplete" [High,Fix committed] https://launchpad.net/bugs/288064
  94 <herb> danilos: We get some input. :) I think it's typically decided a bit higher up the chain of command than the LOSAs
  95 <danilos> herb: and I believe it fluctuates a lot with last-minute fixes, no?
  96 <Ursinha> matsubara_, yes, they are
  97 <danilos> herb: ah, ok
  98 <herb> danilos: it does, but it really shouldn't
  99 <Ursinha> since it landed to edge it's working ok
 100 <danilos> herb: yeah, I see your point
 101 <matsubara_> cool thanks Ursinha 
 102 <danilos> herb: is the date confirmed at least?
 103 <matsubara_> herb: mrevell sent an announcement with dates and estimated down time 
 104 <herb> danilos: My understanding is we're firm on the date.
 105 <herb> matsubara_: ok. cool.
 106 <mrevell> My understand is that we're not really firm on it
 107 <danilos> herb: cool, thanks
 108 <herb> matsubara_: I haven't waded through all my email yet this morning.
 109 <mrevell> Sorry to be a pain but I think there's still some confusion over it
 110 <mrevell> I may yet have to correct my announcement
 111 <herb> mrevell: haha
 112 <matsubara_> oh
 113 <herb> mrevell: ok
 114 <mrevell> i've emailed kiko
 115 <matsubara_> I guess this can be sorted out outside this meeting anyway.
 116 <mrevell> sure
 117 <matsubara_> rockstar: any new about the loggerhead bug?
 118 <matsubara_> s/new/news/
 119 <rockstar> matsubara_, I'm making a note to STRONGLY bring it up again.
 120 <rockstar> With mwhudson on holiday, we haven't moved too much.
 121 <matsubara_> been critical for some time now. how do we handle critical bugs outside of the main launchpad code?
 122 <rockstar> I'll work with beuno and see if we can't get something together.
 123 <rockstar> matsubara_, maybe we should bring it in and make it critical with launchpad-bazaar.
 124 <rockstar> It's not critical for loggerhead, since they never see it.
 125 <herb> rockstar: just to give you some idea, it's the most common cause for nagios alarms for the OSAs.  which means our phones ring at us, which means it really annoys us. doesn't necessarily raise the importance, but definitely adds a story to the problem.
 126 <rockstar> In fact, the bzr playground for gnome never sees it.  It's just us that is affected by it.
 127 <intellectronica> if it can plausibly be dealt with like a critical, maybe it shouldn't be? there's always a risk of debasing that term
 128 <rockstar> herb, I know the false alarm dance.  I'll do what I can to make some noise.
 129 <matsubara_> [action] rockstar to work with beuno re: critical loggerhead (bug 156453)
 130 <MootBot> ACTION received:  rockstar to work with beuno re: critical loggerhead (bug 156453)
 131 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
 132 <herb> rockstar: thanks
 133 <matsubara_> thanks rockstar 
 134 <matsubara_> anything else herb?
 135 <herb> nothing.  thanks
 136 <matsubara_> ok thank you herb 
 137 <matsubara_> moving on
 138 <matsubara_> [topic] * DBA report (DBA contact)
 139 <MootBot> New Topic:  * DBA report (DBA contact)
 140 <stub> All known issues we have had with replication in the staging environment are understood and sorted. I don't see any blockers for switching things on in production this cycle assuming QA doesn't show up anything (in particular, Launchpad APIs).
 141 <stub> DB patches are all landed except for the Job control system (in discussion between the bazaar team and Mark I believe), and data migration to take advantage of the ProductRelease -> Milestone linkage. When the data migration is done will depend on the functionality of the code rolled out (no point migrating the data on rollout or mid cycle if there is not code in place to maintain the migrated data correctly).
 142 <stub> oot.
 143 <matsubara_> thanks stub 
 144 <matsubara_> stub: was an email sent re: the staging db name change?
 145 <stub> Nope.
 146 <stub> So the name of the staging db has changed to lpmain_staging and lpmain_staging_slave
 147 <matsubara_> really?
 148 <matsubara_> isn't launchapd_staging to lpmain_staging only?
 149 <matsubara_> stub: can you email the list about that change?
 150 <matsubara_> thanks stub 
 151 <stub> People can query the slave if they want, but they probably don't want
 152 <matsubara_> stub: all right
 153 <matsubara_> anything else before I close?
 154 <stub> Names will change again at some point in the future (I've learnt more now), but the scripts are working fine atm and no gains for messing with it right now.
 155 <matsubara_> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 156 <matsubara_> stub: ok. let us know once it changes :-)
 157 <rockstar> Thanks matsubara_!
 158 <intellectronica> thanks matsubara_
 159 <matsubara_> #endmeeting
 160 <MootBot> Meeting finished at 09:25.