1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 09:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   5 <matsubara> [TOPIC] Roll Call
   6 <MootBot> New Topic:  Roll Call
   7 <sinzui> me
   8 <gmb> me
   9 <gmb> (in for intellectronica)
  10 * flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
  11 <henninge_> me
  12 <flacoste> me
  13 <bigjools> me
  14 * Ursinha (n=ursula@canonical/launchpad/ursinha) has joined #launchpad-meeting
  15 <Ursinha> erm
  16 <Ursinha> me?
  17 <mrevell> me
  18 <matsubara> stub, herb, henninge?
  19 <matsubara> rockstar: ?
  20 <stub> me
  21 <henninge_> me!
  22 <herb> me
  23 <rockstar> me
  24 <rockstar> Sorry, was ill yesterday, still not at 1[[%
  25 <rockstar> 100%
  26 <matsubara> all right, everyone is here
  27 <matsubara> [TOPIC] Agenda 
  28 <MootBot> New Topic:  Agenda
  29 <matsubara>  * Next meeting
  30 <matsubara>  * Actions from last meeting
  31 <matsubara>  * Oops report & Critical Bugs 
  32 <matsubara>  * Operations report (mthaddon/herb/spm)
  33 <matsubara>  * DBA report (DBA contact)
  34 <matsubara> so next meeting, same time next week?
  35 <matsubara> I guess that's a yes
  36 <matsubara> all right
  37 <matsubara> [topic] * Actions from last meeting
  38 <MootBot> New Topic:  * Actions from last meeting
  39 <matsubara> * rockstar to work with beuno re: critical loggerhead (bug 156453)
  40 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
  41 <matsubara> rockstar: any news about that one?
  42 <rockstar> matsubara, working on it.
  43 <Ursinha> +1
  44 <rockstar> I don't think it's something that's going to happen overnight.
  45 <Ursinha> rockstar, is it really critical?
  46 <rockstar> Ursinha, well, yes.
  47 * henninge has quit (Read error: 110 (Connection timed out))
  48 <matsubara> rockstar: any eta?
  49 <rockstar> I bet the LOSAs think it's psat Critical.
  50 * henninge_ is now known as henninge
  51 <henninge> me again
  52 <rockstar> matsubara, until we're sure what the problem is, I can't give you an ETA.
  53 <matsubara> rockstar: or can you give us regular status report on the bug report itself? critical bugs shouldn't be left like nobody is looking at them
  54 * herb definitely thinks it's critical.
  55 <herb> particularly since I got a page about it right before the meeting. :)
  56 <rockstar> matsubara, I don't really have much to report on it.
  57 <rockstar> It's easy to make it leak memory, it's hard to see where it is leaking memory.
  58 <Ursinha> rockstar, nothing, like partial reports, what are you guys thinking to do
  59 <Ursinha> ?
  60 <Ursinha> oh
  61 <Ursinha> see
  62 <matsubara> rockstar: by looking at the report it seems that it's been stuck since 2008-11-06
  63 <rockstar> matsubara, I can put "Still working on this" every once in a while.  I just don't have specifics, unfortunately.
  64 <matsubara> right
  65 <rockstar> Okay, I'll do that.
  66 <matsubara> thanks rockstar 
  67 <matsubara> [TOPIC]  * Oops report & Critical Bugs 
  68 <MootBot> New Topic:   * Oops report & Critical Bugs
  69 * matsubara hands the mic to Ursinha 
  70 <Ursinha> ok
  71 <Ursinha> mics make me nervous
  72 * bigjools waits for the Karaoke
  73 <Ursinha> have two bugs and one timeout
  74 <Ursinha> matsubara's bug: 300320
  75 <Ursinha> argh
  76 <Ursinha> bug 300320
  77 <ubottu> Launchpad bug 300320 in launchpad-foundations "mailman oops directory permission keep getting reset" [High,New] https://launchpad.net/bugs/300320
  78 <matsubara> that's not critical but it's blocking our work
  79 <Ursinha> so the bot isn't here
  80 <matsubara> I've asked herb to fix the permission
  81 <Ursinha> hahaha
  82 <Ursinha> ok
  83 <Ursinha> so matsubara filed this one
  84 <matsubara> but we need to find out what's resetting the permission every once in a while
  85 <herb> matsubara: trying to do some debugging on that, but will reset the permissions after the meeting.
  86 <Ursinha> matsubara, indeed, just got one of these before the meeting
  87 <matsubara> thanks herb. let me know any news on the report please
  88 <matsubara> flacoste: I'm not sure that's a foundations or registry issue, btw 
  89 <Ursinha> ok
  90 <sinzui> Bug 300183 is critical. I just made it critical
  91 <ubottu> Bug 300183 on http://launchpad.net/bugs/300183 is private
  92 <flacoste> matsubara: registry
  93 <Ursinha> the other is for bugs, bug 300324
  94 <ubottu> Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324
  95 * Ursinha kicks ubottu 
  96 <flacoste> sinzui: you are aware of bug 300320?
  97 <ubottu> Launchpad bug 300320 in launchpad-foundations "mailman oops directory permission keep getting reset" [High,New] https://launchpad.net/bugs/300320
  98 <matsubara> flacoste: I'll re-assign and coordinate with barry and sinzui then if it's a code or LP config problem. thanks
  99 <Ursinha> it's the bug regarding that bug that had its comment 0 deleted accidentally
 100 <gmb> Ursinha: I've just triaged that one.
 101 <gmb> The simple fix is to delete all of bug 297388's subscribers
 102 <ubottu> Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text)
 103 <gmb> and to mark it private
 104 <gmb> hush, ubottu
 105 <sinzui> flacoste: I saw it today. I believed it related install problem that is bug 300183
 106 <ubottu> Bug 300183 on http://launchpad.net/bugs/300183 is private
 107 <gmb> Doing that will make it invisible, to all intents and purposes.
 108 <stub> The OOPS dir permissions are probably being reset by rsync, so the fix is to ensure permissions are set correctly on forster or to tell rsync to make everything locally world readable
 109 <Ursinha> gmb, thanks, it's really annoying and I'm just waiting for some nice bugs guy to solve that for us
 110 <gmb> Ursinha: Ultimately we want to fix bug 297411
 111 <ubottu> Launchpad bug 297411 in malone "Delete a given bug" [Undecided,Triaged] https://launchpad.net/bugs/297411
 112 <Ursinha> hahaha you just caused an oops
 113 <gmb> But it's not easy to do.
 114 * Ursinha kicks ubottu again
 115 <matsubara> thanks stub. 
 116 <Ursinha> gmb, I saw BjornT 's comment
 117 <Ursinha> gmb, but for now we need something to stop this bug to oops
 118 <stub> gmb: If we only hit the red button for spam, we can do it by changing a load of constraints to ON DELETE CASCADE.
 119 <Ursinha> it's filling the summaries with oopses
 120 <gmb> Ursinha: Well, hiding it should do that.
 121 <gmb> Because veeryone will get "you can't access this bug" eros.
 122 <gmb> Erk.
 123 <gmb> Waaaay wrong word, there.
 124 <gmb> *errors
 125 <Ursinha> gmb, hm
 126 <Ursinha> gmb, is it the easiest thing to do?
 127 <gmb> Ursinha: Yes. By far.
 128 <gmb> The trouble is that the Bug table has a lot of dependencies
 129 <gmb> And the dependencies also have a lot of dependencies
 130 <gmb> So we have to find all the things that might depend upon the Bug record before we can DELETE it.
 131 <Ursinha> gmb, well, but do you need that when hiding?
 132 <Ursinha> considering hiding != deleting
 133 <gmb> UPDATE Bug SET private = True WHERE id = 287411;
 134 <gmb> DELETE FROM BugSubscription WHERE bug = 297411;
 135 * Notify: kiko is offline (lem.freenode.net).
 136 <gmb> (yes, the first number was wrong)
 137 <gmb> And that should do it.
 138 <Ursinha> gmb, making it private will prevent privileged people to get the oops?
 139 <matsubara> gmb: can you add that to LaunchpadProductionStatus page and ask the Losas to run it?
 140 <gmb> Good question.
 141 <gmb> matsubara: Sure.
 142 <stub> gmb: But we can ignore a lot of the dependencies - if this is spam protection, those dependencies won't exist.
 143 <matsubara> [action] gmb to add sql to hide bug 297411 as a fix for bug 300324
 144 <MootBot> ACTION received:  gmb to add sql to hide bug 297411 as a fix for bug 300324
 145 <ubottu> Launchpad bug 297411 in malone "Delete a given bug" [Undecided,Triaged] https://launchpad.net/bugs/297411
 146 <ubottu> Launchpad bug 300324 in malone "Bug 297388 OOPSing because had its 0 comment deleted" [High,Triaged] https://launchpad.net/bugs/300324
 147 <gmb> stub: Is there a way to do a DELETE without Postgres erroring on un-deleted dependenices?
 148 <Ursinha> another oops coming..
 149 <matsubara> shit, that action is wrong
 150 <matsubara> [action] gmb to add sql to hide bug 297388 as a fix for bug 300324
 151 <MootBot> ACTION received:  gmb to add sql to hide bug 297388 as a fix for bug 300324
 152 <gmb> Ursinha: If ubottu listens to its own drivel.
 153 <ubottu> Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text)
 154 <ubottu> Error: Could not parse data returned by Launchpad: HTTP Error 500: Internal Server Error (https://launchpad.net/bugs/297388/+text)
 155 <gmb> Damn.
 156 <stub> gmb: No, but we can put up with an oops in that case since an admin is hitting the button when they shouldn't
 157 <Ursinha> arrrrr
 158 <gmb> stub: I'm not quite sure that I'm on your wavelength here. Are you talking about a fix for the OOPS Ursinha raised or for bug 297411 in general?
 159 <ubottu> Launchpad bug 297411 in malone "Delete a given bug" [Undecided,Triaged] https://launchpad.net/bugs/297411
 160 * gmb avoids another OOPS
 161 <stub> gmb: Bug deletion in general (so wrong channel I guess)
 162 <matsubara> Ursinha: anything else?
 163 <gmb> stub: Ah right. Yes, let's talk about that elsewhere and not pollute the meeting.
 164 <Ursinha> thanks gmb :P
 165 <matsubara> I think we can discuss the bug deletion bug in the mailing list or whatever. it's not a simple fix anyway
 166 <Ursinha> matsubara, I have one timeout oops
 167 <Ursinha> OOPS-1055EC45
 168 <Ursinha> I'd like someone to take a look on it
 169 <Ursinha> few occurrences but high sql time, after the rollout
 170 <Ursinha> ubottu, I hate you man
 171 <ubottu> Sorry, I don't know anything about i hate you man
 172 <Ursinha> lol
 173 <Ursinha> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1055EC45
 174 <matsubara> sinzui: ^
 175 <matsubara> can you take a look?
 176 <Ursinha> I have an incredible lag today
 177 <sinzui> yuck
 178 <flacoste> matsubara: could you make the left pane on that page viewport fixed (so that it is always available wherever you are on the oops)?
 179 <flacoste> matsubara: the oops report page i mean
 180 <Ursinha> flacoste, good idea
 181 <matsubara> flacoste: yeah, got that. will do
 182 <matsubara> flacoste: and thanks for the suggestion :-)
 183 <matsubara> [action] ursinha to file a bug about OOPS-1055EC45 and coordinate with Registry to get it fixed
 184 <Ursinha> this is all from me
 185 <MootBot> ACTION received:  ursinha to file a bug about OOPS-1055EC45 and coordinate with Registry to get it fixed
 186 <matsubara> let's move on then
 187 <matsubara> thanks Ursinha and everyone else
 188 <matsubara> [topic] * Operations report (mthaddon/herb/spm)
 189 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 190 <sinzui> matsubara: flacoste I can pass this to Edwin because I think he has  day for this.
 191 <herb>  2008-11-19: Rolled out r7316 to production.
 192 <herb> - Bugs #118625 and #156453 continue to be a problem causing us to restart codebrowse on a number of ocassions this week. As we discussed earlier.
 193 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
 194 <herb> - We can probably mark bug #247227 as Fix Released since the app servers have been stable for several weeks now.
 195 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
 196 <ubottu> Bug 247227 on http://launchpad.net/bugs/247227 is private
 197 <herb> - Do we have a decision on if/when there will be a re-roll?
 198 <Ursinha> thanks guys
 199 <matsubara> thanks sinzui 
 200 <matsubara> Ursinha: add that the report ^ :-)
 201 <matsubara> s/the/to the/
 202 <Ursinha> matsubara, will do
 203 <matsubara> herb: I talk to kiko about it today and let you know
 204 <matsubara> about the re-roll I mean
 205 <herb> matsubara: ok
 206 <Ursinha> I'll ask him to join
 207 * kiko-fud (n=kiko@canonical/launchpad/kiko) has joined #launchpad-meeting
 208 <kiko-fud> yes dears
 209 <Ursinha> <herb> - Do we have a decision on if/when there will be a re-roll?
 210 <kiko-fud> tonight 22:00 UTC if at all
 211 <kiko-fud> I have spoken
 212 <matsubara> cool. thanks kiko-fud 
 213 <Ursinha> ok, great
 214 <matsubara> thanks herb 
 215 <Ursinha> thanks kiko-fud
 216 <matsubara> [topic] * DBA report (DBA contact)
 217 <MootBot> New Topic:  * DBA report (DBA contact)
 218 <herb> thanks Ursinha, matsubara, kiko-fud
 219 <herb> et al
 220 * matsubara hands the mic to stub 
 221 <stub> Our replica production database built happily, and maintaining the replica does not seem to be adding noticable load. My worries about diskspace where thankfully unfounded.
 222 <stub> We may find that chokecherry's load is too high when we start the appservers using the replica during staging rebuilds. If this turns out to be the case, we have the choice of not using the replica or moving staging somewhere else.
 223 <stub> oot
 224 <matsubara> stub: and the plan is to have that on production next cycle?
 225 <stub> Its just a config change - we can switch it on whenever.
 226 <matsubara> cool.
 227 <stub> I've got a branch prepared with the config changes and kiko knows already
 228 <matsubara> Ursinha: ^ got your answer for waht you asked me yesterday? :-0
 229 <matsubara> :-)
 230 <matsubara> great
 231 <matsubara> thanks stub 
 232 <Ursinha> matsubara, yes :)
 233 <matsubara> anything else before I close?
 234 <flacoste> matsubara: i'll land that branch
 235 <Ursinha> thanks matsubara and statik
 236 <Ursinha> errr
 237 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 238 <matsubara> #endmeeting 
 239 <Ursinha> and stub
 240 <Ursinha> sorry statik