1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 09:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <henninge> me
   8 <intellectronica> me
   9 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
  10 <herb> me
  11 <Ursinha> me
  12 <stub> me
  13 <matsubara> sinzui, flacoste?
  14 <rockstar> me
  15 <flacoste> me
  16 <matsubara> bigjools: ?
  17 <bigjools> meeeeeeee
  18 <sinzui> m,e
  19 <sinzui> me
  20 <matsubara> all right, everyone is here
  21 <matsubara> [TOPIC] Agenda 
  22 <MootBot> New Topic:  Agenda
  23 * sinzui hates windows that steal focus
  24 <matsubara>  * Next meeting
  25 <matsubara>  * Actions from last meeting
  26 <matsubara>  * Oops report & Critical Bugs 
  27 <matsubara>  * QA pending items
  28 <matsubara>  * Operations report (mthaddon/herb/spm)
  29 <matsubara>  * DBA report (DBA contact)
  30 <matsubara> [TOPIC] * Next meeting
  31 <MootBot> New Topic:  * Next meeting
  32 <matsubara> same time next week?
  33 <matsubara> I'll be on vacation, so Ursinha will take over
  34 * bigjools is out for 3 weeks
  35 <Ursinha> o/
  36 <matsubara> [action] matsubara to add himself and bigjools to the apologies list
  37 <MootBot> ACTION received:  matsubara to add himself and bigjools to the apologies list
  38 <matsubara> bigjools: can you arrange for someone in soyuz to cover for you?
  39 <bigjools> matsubara: yep
  40 <matsubara> bigjools: thanks
  41 <matsubara> [TOPIC] * Actions from last meeting
  42 <MootBot> New Topic:  * Actions from last meeting
  43 <matsubara>  * matsubara email stub asking for the DBA report
  44 <matsubara>  * matsubara to talk with gmb and add sql to hide bug 2-9-7-3-8-8 as a fix for bug 300324
  45 <matsubara>  * rockstar to talk with code team about their pending QA items
  46 <matsubara>  * henninge  to talk with translation team about their pending QA items
  47 <matsubara>  * Ursinha to talk with bigjools about one account on dogfood machine to do some testing
  48 <matsubara>  * rockstar to handle bug 271879 today (12-04)
  49 <matsubara>  * Ursinha to watch timeouts in the next days to see if they continue to happen this often
  50 <matsubara>  * rockstar to manage having something reported on bug 118625 and bug 156453
  51 <matsubara>  * Ursinha to watch for strange OOPSes on edge in the next days
  52 <ubottu> Error: Launchpad bug 2 could not be found
  53 <ubottu> Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324
  54 <ubottu> Bug 271879 on http://launchpad.net/bugs/271879 is private
  55 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
  56 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
  57 <Ursinha> oh yeah
  58 <Ursinha> I've opened a RT to get access to mawson
  59 <rockstar> matsubara, first item is talked about everyday now.
  60 <matsubara> I did both of mine and gmb added the query to the the LPPS page effectively hiding the problematic bug
  61 <Ursinha> matsubara, great
  62 <Ursinha> I've watched for the oopses and nothing that bizarre came up
  63 <rockstar> matsubara, the fix for the timeout bug is approved, but we need to do some query testing.
  64 <matsubara> thanks rockstar. Is the daily nagging helping QA testing?
  65 <rockstar> matsubara, my team doesn't like nagging, but they're getting to it.
  66 <matsubara> Ursinha: anything unusual about the strange OOPS and timeouts?
  67 <Ursinha> matsubara, nothing that's not reported and being taken care of
  68 <matsubara> henninge: how's the daily nagging about pending QA items going? is it helping your team?
  69 <henninge> matsubara: Well, currently all items have been tested that could have been.
  70 <henninge> matsubara: danilo is at UDS right now and jtv doesn't have anything else to test.
  71 <matsubara> great. thanks henninge 
  72 <henninge> matsubara: we'll keep reminding ourselves ...
  73 <matsubara> cool
  74 <matsubara> moving on
  75 <matsubara> [TOPIC] * Oops report & Critical Bugs 
  76 <MootBot> New Topic:  * Oops report & Critical Bugs
  77 <Ursinha> okay
  78 <matsubara> Ursinha: do you want me to do it?
  79 <matsubara> all right, go ahead
  80 <Ursinha> matsubara, just saw sinzui was talking to salgado about bug 304113
  81 <ubottu> Launchpad bug 304113 in launchpad-registry "Milestone:+index page timing out" [High,Triaged] https://launchpad.net/bugs/304113
  82 <Ursinha> oops, this wasn't for you matsubara, sorry :)
  83 <sinzui> Ursinha: we will start working on it in a few hours
  84 <Ursinha> sinzui, great, thanks
  85 <Ursinha> henninge, can you ask someone on translations to set priority and importance on bug 302798, please?
  86 <ubottu> Launchpad bug 302798 in rosetta "Timeout on +translate page" [Undecided,New] https://launchpad.net/bugs/302798
  87 <henninge> Ursinha: yes
  88 <Ursinha> henninge, thank you :)
  89 <matsubara> priority and importance ?
  90 <Ursinha> erm
  91 <matsubara> we have only importance :-)
  92 <henninge> status and importance
  93 <Ursinha> status and importance, sorry :)
  94 <henninge> Ursinha: I knew what you meant ... ;)
  95 <matsubara> anything else Ursinha ?
  96 <Ursinha> rockstar, you
  97 <rockstar> Me?!
  98 <Ursinha> rockstar, do you know anything about bug 305367
  99 <ubottu> Bug 305367 on http://launchpad.net/bugs/305367 is private
 100 <Ursinha> ?
 101 <Ursinha> grrr
 102 <Ursinha> it's a critical one
 103 <Ursinha> the only one we have
 104 <Ursinha> it's really critical?
 105 <rockstar> Ursinha, I don't think they are critical.
 106 <rockstar> activereviews is for your own code.
 107 <Ursinha> rockstar, stub marked it critical
 108 <rockstar> I'm working on the pages to generate a list of reviews you need to vote on.
 109 <Ursinha> rockstar, right.
 110 <rockstar> I'm going to bring it down from critical, anh talk with stub.
 111 <stub> I marked it critical because I couldn't see a work around and might lose things. Feel free to point me at a work around :)
 112 <Ursinha> rockstar, thank you very much
 113 <rockstar> stub, no work around.
 114 <rockstar> stub, I think your people should be contacting you to tell you about them.
 115 <Ursinha> I'm done after that
 116 <rockstar> That should be the workaround right now.
 117 <stub> rockstar: Not really an option with the db review windows - I'll use a wiki page next cycle.
 118 <rockstar> stub, next cycle you'll have the page you're looking for.  I'm working on it now.
 119 <rockstar> But activereviews is for seeing the code you've proposed for review and its status.
 120 <stub> Ok.
 121 <matsubara> all right, moving on.
 122 <matsubara> thanks everyone
 123 <Ursinha> thanks guys
 124 <matsubara> [TOPIC] * QA pending items
 125 <MootBot> New Topic:  * QA pending items
 126 <Ursinha> right
 127 <Ursinha> so we have a lot of items to be tested
 128 <rockstar> We have too many still, and the ones that are getting done aren't getting recorded.
 129 <rockstar> :/
 130 <matsubara> why not?
 131 <Ursinha> rockstar, this is a problem
 132 <rockstar> Ursinha, I'll talk with thumper about it.
 133 <Ursinha> rockstar, you'll lost yourselves with that
 134 <Ursinha> rockstar, okay. I'm on my late shift today, will talk to thumper as weel
 135 <Ursinha> *well
 136 <matsubara> rockstar: what's the issue about recording what's been tested?
 137 * salgado is now known as salgado-lunch
 138 <rockstar> matsubara, no idea.  I know some of these things have been tested.
 139 <matsubara> rockstar: right, so it's just a matter of updating the wiki page or something else?
 140 <rockstar> matsubara, wiki page.
 141 <matsubara> I see. if the process to update the test plan page is too troublesome we need to find a better way. suggestions welcome.
 142 <rockstar> matsubara, also, this week has been abnormally busy with the ec2 stuff.  It might be that recording it isn't at the front of their minds.
 143 <rockstar> matsubara, I need to think about that.
 144 <intellectronica> fwiw there's quite a lot of talk about using launchpad itself to track qa. if anyone has good ideas please let them be known
 145 <Ursinha> rockstar, thanks.
 146 <Ursinha> intellectronica, indeed
 147 <rockstar> intellectronica, yea, I was going to think about that.
 148 <rockstar> The Ubuntu guys might be happy to have qa handled in Lanuchpad.
 149 <rockstar> also, Launchpad.
 150 <intellectronica> so i hear, yes
 151 <Ursinha> any suggestions/ideas are very welcome
 152 <matsubara> that's probably something that will happen in the future, until we get to the pojnt we can use LP to track QA, we should be updating the wiki page
 153 <intellectronica> of course
 154 <matsubara> I know that's far from perfect but don't let that stop the QA. 
 155 <matsubara> one thing you could do is email Ursinha and I all items tested and we'll take care of updating the test plan
 156 <Ursinha> and I'm going to repeat myself again, but, please, ping us if we can do something
 157 <Ursinha> like that matsubara said
 158 <matsubara> until we find a better way to deal with this
 159 <bigjools> Ursinha: halp
 160 <matsubara> thanks
 161 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
 162 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 163 <Ursinha> bigjools, as soon as a have access to mawson, will be glad too. but if there's something I could do now, let me know
 164 <herb> * 2008-12-10 - Cherry picked r7386 enabling replication to production and cherry picked r7419 to edge.
 165 <Ursinha> s/as a/as I/
 166 <herb> * Bugs #118625 and #156453 continue to be a problem despite a patch and the addition of the pyrex package.
 167 <herb> * Had an issue with the branch scanner where a user's subscription to a particularly large branch was causing the branchscanner to hang. See bug #242807
 168 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
 169 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
 170 <herb> * Had to kill off a bunch of old processes on the code hosting server. This seems to be happening a bit more often than once a week now. See bug #260171
 171 <ubottu> Launchpad bug 242807 in ubuntu "my ipod nano is not showing up when plugged in" [Low,Invalid] https://launchpad.net/bugs/242807
 172 <ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
 173 <herb> hmm
 174 <matsubara> ipod nano?
 175 <matsubara> hehe
 176 <Ursinha> lol
 177 <rockstar> 242807 doesn't seem to correlate.
 178 <herb> something tells me I typoed at least one of those.
 179 <herb> should be bug #252807
 180 <ubottu> Bug 252807 on http://launchpad.net/bugs/252807 is private
 181 <rockstar> herb, there are many folken at UDS right now trying to figure out why loggerhead sucks.
 182 <matsubara> herb: btw, did you find out why staging is down?
 183 <rockstar> It's disappointing that it's still having issues.
 184 <matsubara> hmm it's not down anymore, so why staging was down? :-)
 185 <herb> matsubara: the old lp process didn't exit and so the new process couldn't bind to the port after the restore.
 186 <Ursinha> herb, iirc it was the second time this week
 187 <matsubara> herb: ok. thanks for chasing
 188 <Ursinha> the other time happened for the same reason?
 189 <herb> Ursinha: 3rd or 4th time in the last 10 days.
 190 <herb> Ursinha: different reasons the last couple of times. it had been hanging in the middle of startup.
 191 <Ursinha> hm, right.
 192 <rockstar> herb, was it the librarian?
 193 <herb> rockstar: running.  but didn't really look at it in too much detail.
 194 <matsubara> herb: if it happens again, could you file a bug about it please and let me know?
 195 <rockstar> herb, I've noticed the librarian doesn't close cleanly on my dev machine.
 196 <herb> matsubara: sure. I can't provide much detail on the problems I've seen over and over though.  stracing the startup just shows a wait call.
 197 <herb> :(
 198 <matsubara> thanks herb
 199 <matsubara> [TOPIC] * DBA report (DBA contact)
 200 <MootBot> New Topic:  * DBA report (DBA contact)
 201 <stub> The losas altered the production appserver configs so they start talking to the production replica where possible. This is working well... possibly too well.
 202 <stub> Most of the load is now on the slave, with the production box pretty much idling. We can push some of the
 203 <stub> load back to the main box by reverting a few of the appserver configs, but we should probably leave it be until we have the reverse web proxy caching anonymous requests to see how this affects the load distribution.
 204 <stub> I will be running the script to open Jaunty translations with Jeroen tomorrow. This should be the last time it is a heavy weight operation, as translation inheritence is scheduled to be implemented before the next opening. No ETAs on the run time completion yet - I don't fully trust them given the extra overhead we have now on inserts thanks to replication.
 205 <stub> DB patches for this cycle are all either landed or rejected, except for the per language translation policies link with is blocked on discussions upstairs.
 206 <stub> (^-^)
 207 <bigjools> stub: did Celso's get accepted or rejected?
 208 <stub> Which one?
 209 <bigjools> the licensing meta-data one
 210 <stub> Oh... licencing sourcepackagereleae. Rejected. Needs mucho discussion.
 211 <bigjools> blah
 212 <matsubara> all right, anything else for stub?
 213 <matsubara> thanks stub 
 214 <matsubara> anything else before I close?
 215 <matsubara> 5
 216 <matsubara> 4
 217 <matsubara> 3
 218 <matsubara> 2
 219 <matsubara> 1
 220 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 221 <Ursinha> lol
 222 <matsubara> #endmeeting