1 <MootBot> Meeting started at 10:00. The chair is matsubara.
   2 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   3 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <rockstar> me
   8 <danilos> me
   9 <Ursinha> me
  10 <stub> me
  11 <gary_poster> me
  12 <gary_poster> (for flacoste)
  13 <matsubara> hi gary_poster
  14 <gary_poster> hi :-)
  15 <matsubara> herb, 
  16 <herb> me
  17 <matsubara> bigjools, hi
  18 <matsubara> sinzui, hi
  19 <sinzui> me
  20 <matsubara> ok, apologies from the Bugs team, as they're sprinting
  21 <matsubara> eveyrone else is here.
  22 <bigjools> me
  23 <matsubara> [TOPIC] Agenda 
  24 <MootBot> New Topic:  Agenda
  25 <matsubara>  * Actions from last meeting
  26 <matsubara>  * Oops report & Critical Bugs 
  27 <matsubara>  * Operations report (mthaddon/herb/spm)
  28 <matsubara>  * DBA report (stub)
  29 <matsubara> [TOPIC] * Actions from last meeting
  30 <MootBot> New Topic:  * Actions from last meeting
  31 <matsubara>   * Ursinha to file a bug about OOPS-1159EA69 and coordinate a fix with rockstar
  32 <matsubara>    * https://bugs.edge.launchpad.net/launchpad-bazaar/+bug/345415
  33 <matsubara>   * Ursinha to file a bug about OOPS-1166F631 and assign to intellectronica
  34 <matsubara>    * https://launchpad.net/bugs/341766
  35 <matsubara>   * sinzui to request CP for bugs 341085 and 341117
  36 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1166F631
  37 <ubottu> Ubuntu bug 345415 in launchpad-bazaar "NoSuchDistroSeries OOPS in resolve_lp_path" [Undecided,New]
  38 <ubottu> Bug 341085 on http://launchpad.net/bugs/341085 is private
  39 <ubottu> Ubuntu bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Undecided,Won't fix]
  40 <matsubara> ok, Ursinha did her stuff
  41 * sinzui hangs head in shame
  42 <matsubara> sinzui, did you request those CPs?
  43 <sinzui> matsubara: I will do that within the hour
  44 <matsubara> thanks sinzui
  45 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  46 <MootBot> New Topic:  * Oops report & Critical Bugs
  47 <matsubara> Ursinha, go ahead
  48 <Ursinha> have only one bug, sinzui: bug 340058
  49 <ubottu> Launchpad bug 340058 in launchpad-registry "Sometimes people/team pages don't load" [Undecided,New] https://launchpad.net/bugs/340058
  50 <Ursinha> I'm experiencing that sometimes, it's annoying
  51 <Ursinha> the other bug I have is for bugs: [action] talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766
  52 <ubottu> Launchpad bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Undecided,Won't fix] https://launchpad.net/bugs/341766
  53 <sinzui> Ursinha: do you browse with maps on or off?
  54 <matsubara> [action] talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766
  55 <MootBot> ACTION received:  talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766
  56 <Ursinha> sinzui: on
  57 <Ursinha> sinzui: but it's random
  58 <sinzui> Ursinha: I have experienced that once when I discovered I had bad connectivity to google.
  59 <Ursinha> sometimes it happens
  60 <Ursinha> hmmm
  61 <matsubara> I couldn't reproduce that one
  62 <danilos> sinzui, Ursinha: that should be easy to test, if it means no connectivity to google
  63 <sinzui> well it was half a day. I wrote hide google maps to keep working
  64 <matsubara> I wonder if it might be an firefox extension thing
  65 <danilos> just adding 127.0.0.1 whatever-google.com to /etc/hosts could emulate it :)
  66 <gary_poster> heh, good idea :-)
  67 <matsubara> and try it with a clean firefox profile
  68 <Ursinha> sinzui: I can do this testing and report on the bug
  69 <matsubara> [action] ursinha to debug 340058 and results back to the report
  70 <MootBot> ACTION received:  ursinha to debug 340058 and results back to the report
  71 <Ursinha> awesome
  72 <Ursinha> no critical bugs
  73 <Ursinha> so that's it
  74 <sinzui> Ursinha: thanks. please do. There is not enough information for me to say this is a bug
  75 <matsubara> ok, thanks Ursinha, sinzui
  76 <matsubara> moving on
  77 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
  78 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
  79 <herb> 2009-03-13 - Cherry picked r7939 to the codehosting server as a fix for bug #260171.
  80 <ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
  81 <herb> 2009-03-14 - The librarian box hung and didn't come back cleanly when power cycled. The librarian service was unavailable for about an hour and a half.
  82 <herb> 2009-03-17 - Cherry picked r7942 to the scripts server as a fix for bug #337658.
  83 <ubottu> Launchpad bug 337658 in rosetta "Disable success notifications for published uploads" [High,Fix released] https://launchpad.net/bugs/337658
  84 <herb> Bug #156453 and bug #118625 continue to be a problem for us.
  85 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
  86 <herb> Bug #45419 continues to be  a pain point for the LOSAs.
  87 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
  88 <ubottu> Launchpad bug 45419 in launchpad-foundations "Launchpad needs a way of easily flagging spam" [High,Triaged] https://launchpad.net/bugs/45419
  89 <herb> There are currently 2 requested cherry picks pending approval/rejection, and have been that way for more than a week. Can we get those approved or rejected?
  90 <danilos> I'd also mention that staging is under a special regime, for anyone who hasn't seen my email
  91 <herb> danilos: good point. thanks
  92 <gary_poster> Is it appropriate to ask now about the status of [rt.admin.canonical.com #32927] launchpad buildbot master in data center ?
  93 <mthaddon> gary_poster: it's in the "setting up the EC2 account" stage
  94 <gary_poster> mthaddon: cool, thanks
  95 <herb> gary_poster: you can ask. I don't have any input for you though.  I'll can look into it for you.
  96 <danilos> matsubara: who's the soyuz QA contact?
  97 <matsubara> bigjools, ^
  98 * bigjools waves
  99 <gary_poster> herb: cool
 100 <herb> or mthaddon can tell you where we stand. :p
 101 <gary_poster> :-)
 102 <danilos> bigjools: the two cherrypick requests are from cprov, just fyi :)
 103 * salgado is now known as salgado-afk
 104 <matsubara> bigjools, kiko/flacoste needs to approve those. can you sort it out with one of them?
 105 <bigjools> matsubara: talk to me after the meeting about this please
 106 <herb> I think that's it from us unless there are questions.
 107 <matsubara> [action] bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-)
 108 <MootBot> ACTION received:  bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-)
 109 <matsubara> bigjools, sure
 110 <matsubara> rockstar, do you have any news about the LH thing?
 111 <rockstar> matsubara, nothing definitive.
 112 <matsubara> rockstar, still chasing the root cause?
 113 <rockstar> matsubara, basically.  It's voodoo at this point.
 114 <matsubara> :-(
 115 <matsubara> well, nothing much we can do here then...
 116 <matsubara> let's move on. thanks herb and everyone else
 117 <matsubara> [TOPIC] * DBA report (stub)
 118 <MootBot> New Topic:  * DBA report (stub)
 119 <stub> The new slave is online and ticking away happily. Its lonely though - LOSAs should configure some of the app servers to talk to it soon. Batch jobs should consider making use of slave databases as we now have spare capacity there.
 120 <stub> It will be interesting to see how the new 16 core box compares to the existing 8 core boxes. Not many people get to play with DB setups in the same league, so we are in somewhat uncharted territory. Previous anecdotal evidence had the twin core Opterons outperforming the quad core Xeons, but that was a while ago and it is a moving target.
 121 <stub> We still need a real load balancer to distribute connections to the slaves. What we have is fine though until the batch jobs start making heavy use of the slaves.
 122 <stub> oot.
 123 <matsubara> all right. questions for stub?
 124 <matsubara>  thanks stub 
 125 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 126 <matsubara> #endmeeting 
 127 <MootBot> Meeting finished at 10:21.
 128 <Ursinha> thanks matsubara and MootBot
 129 <gary_poster> thanks
 130 <rockstar> matsubara, I had a point to bring up about the format.
 131 * gary_poster has quit ()
 132 <matsubara> rockstar, sorry, what format?
 133 <rockstar> Oops Report / Critical Bugs should be condensed down to Critical bug status
 134 <mthaddon> rockstar: what do you mean by "it's voodoo at this point"? is it not reproducible locally (setup an instance with a large branch history and browse it)?
 135 <danilos> matsubara: thanks
 136 <rockstar> Often, when OOPS reports are referenced, I haven't seen them before, and so I'm not prepared really to make a full comment.
 137 <rockstar> mthaddon, I mean "We shook this part of loggerhead, and fixed its performance.  Did it work?  Do I need to make a sacrifice to Apollo?"
 138 <mthaddon> rockstar: but presumably you're still able to reproduce the problem locally?
 139 <matsubara> rockstar, right. I asked Ursinha to bring up only bugs instead oopses, that way we keep the discussion in the report
 140 <rockstar> matsubara, also, we should know about critical bugs before the meeting, so that we have something to report.
 141 <matsubara> and when we deal with newly filed bug reports re: unseen oopses, the meeting section would be a topic to remind you to investigate that report further
 142 <rockstar> mthaddon, nominally.  We can see the memory climbing, but we don't really get the "hang" which I'm presuming is a separate bug.
 143 <rockstar> mthaddon, the "hang" is probably specific to the way Launchpad serves branches, so we'll probably need to do some sleuthing there.
 144 <mthaddon> rockstar: so it continues to respond quickly for you? The "hang" is basically just "it takes more than 20 seconds to respond and our proxy connection times out
 145 <matsubara> rockstar, we kinda expect you to know about critical bugs under your teams responsibility
 146 <mthaddon> s/out/out"/
 147 <rockstar> mthaddon, but mwhudson has landed quite a few branches to make loggerhead load only what's used when it's used.
 148 <mthaddon> rockstar: I'll chat with mwhudson when he comes online later - I haven't asked him specifically about what he can and can't reproduce - thx
 149 <matsubara> rockstar, so basically, what I'm saying is that the QA contact should check the critical bugs for their team before the meeting.
 150 <rockstar> matsubara, except that this is the first thing I do in the morning.
 151 <rockstar> :/
 152 <matsubara> rockstar, hmm that's a fair point.
 153 <rockstar> matsubara, but basically, I think it should be treated more like an action
 154 <rockstar> "rockstar to report on critical bug 123456"
 155 <ubottu> Launchpad bug 123456 in xine-lib "podcast crashes amarok" [Undecided,Fix released] https://launchpad.net/bugs/123456
 156 <matsubara> I'd suggest check those criticals the day before then
 157 <rockstar> ubottu, sofa king we todd did
 158 <ubottu> Error: I am only a bot, please don't think I'm intelligent :)
 159 <matsubara> rockstar, and for new things that you're not prepared for, I think what you're suggesting is perfectly fine
 160 <rockstar> matsubara, okay, great.
 161 <matsubara> rockstar, thanks for the suggestions.