1 <Ursinha> #startmeeting
   2 <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   3 <Ursinha> [TOPIC] Roll Call
   4 <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
   5 <MootBot> Meeting started at 10:00. The chair is Ursinha.
   6 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   7 <MootBot> New Topic:  Roll Call
   8 <sinzui> me
   9 <Ursinha> me
  10 <Ursinha> !
  11 <herb> me
  12 <rockstar> ni!
  13 <jtv> me
  14 <Ursinha> lol
  15 <Ursinha> I demand a SHRUBBERY!
  16 <mars> me
  17 <Ursinha> matsubara, hi
  18 <jtv> (the rest of the Translations team is out on vacation.  Presumably I'll get a T-shirt)
  19 <Ursinha> lol
  20 <Ursinha> bigjools, hi
  21 <bigjools> me, still on TL call
  22 <Ursinha> okay, nothing to soyuz today
  23 <Ursinha> from oops land
  24 <Ursinha> stub, hi
  25 <matsubara> me
  26 <stub> me
  27 <Ursinha> matsubara, welcome back
  28 <matsubara> thanks Ursinha
  29 <Ursinha> [TOPIC] Agenda
  30 <Ursinha>  * Actions from last meeting
  31 <Ursinha>  * Oops report & Critical Bugs & Broken scripts
  32 <Ursinha>  * Operations report (mthaddon/herb/spm)
  33 <Ursinha>  * DBA report (stub)
  34 <MootBot> New Topic:  Agenda
  35 <Ursinha> [TOPIC] * Actions from last meeting
  36 <Ursinha>     * matsubara to chase rockstar about failure on updatebranches script
  37 <Ursinha>     * stub to get RC for branch that fixes bug 403283
  38 <Ursinha>         * landed in r8319
  39 <Ursinha>     * ursinha do file bug for OOPS-1300XMLP5
  40 <Ursinha>         * filed https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403606
  41 <Ursinha>     * Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting
  42 <Ursinha>         * we're having less errors, Salgado will try to fix bug 61171
  43 <Ursinha>     * mars to take a look at bug 354593
  44 <Ursinha>     * matsubara to chase salgado about people pruning script
  45 <MootBot> New Topic:  * Actions from last meeting
  46 <ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
  47 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
  48 <ubottu> Launchpad bug 403606 in launchpad-foundations "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New]
  49 <ubottu> Bug 61171 on http://launchpad.net/bugs/61171 is private
  50 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
  51 <jtv> Ursinha: We're having "fewer" errors :-P
  52 <stub> The person pruner is running on production. It will take 3 or 4 weeks to complete.
  53 <matsubara> Ursinha, I haven't had time to do that this week as I was on vacation
  54 <Ursinha> thanks jtv :P
  55 <Ursinha> matsubara, okay
  56 <matsubara> please, re-add to the list and I'll chase
  57 <Ursinha> [action] matsubara to chase rockstar about failure on updatebranches script
  58 <mars> stub, wow
  59 <MootBot> ACTION received:  matsubara to chase rockstar about failure on updatebranches script
  60 <matsubara> Ursinha, the other one about the pruning script too
  61 <Ursinha> [action] matsubara to chase salgado about people pruning script
  62 <MootBot> ACTION received:  matsubara to chase salgado about people pruning script
  63 <stub> Chase what? it is running.
  64 <Ursinha> but what stub said?
  65 <Ursinha> yes, yes
  66 <rockstar> Yes, what stub said.
  67 <Ursinha> matsubara, ^
  68 <salgado> stub, don't we need to pass --experimental to it?
  69 <stub> It is - I filed an rt and chased it through with spm.
  70 <stub> And monitoring too
  71 <matsubara> ok, so that's sorted then :-)
  72 <Ursinha> [action] remove last matsubara item as stub already reported it's ok]
  73 <MootBot> ACTION received:  remove last matsubara item as stub already reported it's ok]
  74 <matsubara> :-)
  75 <matsubara> thanks Ursinha
  76 <Ursinha> matsubara, np :)
  77 <Ursinha> mars, did you have the time to look at bug 354593?
  78 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
  79 * Ursinha wonders if mars is on the TL call as well
  80 <mars> nope
  81 <rockstar> Why is there a TL call this early?
  82 <Ursinha> *whew*
  83 <mars> I did not have a chance to address it
  84 <Ursinha> oops section is all foundations today
  85 <rockstar> I doubt my TL is on that TL call...
  86 <bigjools> he's not
  87 <Ursinha> well, mars, can you do that then, please? :)
  88 <mars> stub, would you be able to take the SSO bug?  Or find someone who has time to address it?
  89 <stub> The branding one?
  90 <Ursinha> stub, yes sir
  91 <mars> yes
  92 <Ursinha> stub, bug 354593
  93 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
  94 <stub> I can try. I haven't done UI stuff for a long time though so it will be slow.
  95 <Ursinha> mars, ^
  96 <mars> Ursinha, ?
  97 <mars> stub, I can give you a hand with it
  98 <Ursinha> okay then
  99 <Ursinha> [action] stub to give a try on bug 354593 with mars help if needed
 100 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
 101 <MootBot> ACTION received:  stub to give a try on bug 354593 with mars help if needed
 102 <Ursinha> moving on
 103 <Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
 104 <Ursinha> mars, I have two bugs and one oops for foundations
 105 <Ursinha> mars, can you triage bug 403606, and take a look at it, if possible?
 106 <Ursinha> mars, also, OOPS-1307J16 shows an AssertionError without any description
 107 <Ursinha> and finally
 108 <Ursinha> mars, do you know if someone could have some time to fix bug 310818? we had some weird timeouts today and the oops report was borked
 109 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
 110 <ubottu> Launchpad bug 403606 in launchpad-foundations "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
 111 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
 112 <ubottu> Launchpad bug 310818 in launchpad-foundations "Oops report does not always log timed-out query" [High,Triaged] https://launchpad.net/bugs/310818
 113 <mars> Ursinha, I'll have to ask the team
 114 <Ursinha> mars, about what exactly? :)
 115 <Ursinha> btw, thanks for helping with the Unicode issues debugging last week, was very useful
 116 <mars> Ursinha, to see who on foundations has time to address the issue - I would guess gary or stub, but I don't know how heavily committed they are at the moment
 117 <Ursinha> mars, the bug 310818?
 118 <ubottu> Launchpad bug 310818 in launchpad-foundations "Oops report does not always log timed-out query" [High,Triaged] https://launchpad.net/bugs/310818
 119 <stub> I was going to look at it but someone designated me victim for the SSO exception stuff ;)
 120 <Ursinha> haha
 121 <gary_poster> Heh, It feels like I have a backlog to kingdom come, and I keep getting more. :-)  But if this is urgent, then it's urgent
 122 <stub> I can probably do both - what should be done first Ursinha?
 123 <Ursinha> stub, the oops one, I'd suggest
 124 * mrevell (n=matthew@canonical/launchpad/mrevell) has joined #launchpad-meeting
 125 <Ursinha> as I said earlier we had some weird timeouts and it would be nice to be able to debug them if they happen again
 126 <Ursinha> [action] stub to fix bug 310818
 127 <mars> Ursinha, sorry, I was reading the Expat error one.  That sounded like something gary would be able to address, since it starts in the heart of Zope, and has to do with exception bubbling through the architecture
 128 <ubottu> Launchpad bug 310818 in launchpad-foundations "Oops report does not always log timed-out query" [High,Triaged] https://launchpad.net/bugs/310818
 129 <MootBot> ACTION received:  stub to fix bug 310818
 130 <Ursinha> mars, right. gary_poster, want to fix that? :)
 131 <mars> gary_poster, ^ does that make sense?
 132 <gary_poster> looking
 133 <mars> Ursinha, I'll look at OOPS-1307J16
 134 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
 135 <Ursinha> [action] mars to take a look at OOPS-1307J16
 136 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
 137 <MootBot> ACTION received:  mars to take a look at OOPS-1307J16
 138 <Ursinha> thanks mars 
 139 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1307J16
 140 <Ursinha> mars, I'd open a bug but had no clue about what happened
 141 <gary_poster> Ursinha, not precisely clear on the goal but we can talk later.  I'm guessing this is low priority.  I'll take it.
 142 <gary_poster> (bug 403606)
 143 <ubottu> Launchpad bug 403606 in launchpad-foundations "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
 144 <Ursinha> thanks gary_poster 
 145 <Ursinha> gary_poster, the point is that it fills the oops reports with those
 146 <Ursinha> it would be great to get rid of them
 147 <gary_poster> and they just mean that somebody is sending us bad XMLRPC, afaict, and we don't want to care?
 148 <mars> gary_poster, it would be nice if the ExpatError was, say, turned into a 400 HTTP status code
 149 <mars> right
 150 <Ursinha> creating a section on the summary or moving them to dev/null was told to be not a good idea (I agree)
 151 <gary_poster> ack, ok.
 152 <Ursinha> gary_poster, do you think it's to painful to fix?
 153 <Ursinha> *too
 154 <Ursinha> [action] Ursinha to learn to type
 155 <MootBot> ACTION received:  Ursinha to learn to type
 156 <gary_poster> Ursinha: It will involve changing zope publication machinery.  Doing so will mean either hacking our zope tree, which we are really trying not to do; or migrating to a newer version of the publication machinery, which should wait on the Zope-buildbot work that I keep not finishing, and then will be a migration exercise, possibly accompanied with a negotiate-with-upstream exercise.
 157 <gary_poster> lol
 158 <gary_poster> so...
 159 <gary_poster> the only quick and easy fix is the hack
 160 <Ursinha> I see.....
 161 <gary_poster> that I would be hoping to eliminate RSN
 162 <gary_poster> when I move all the zope stuff to eggs
 163 <Ursinha> gary_poster, so, a very temporary hack?
 164 <gary_poster> so I'm happy to have that be a bug, but I'd like it to be a back-burner bug, myself
 165 <gary_poster> right
 166 <gary_poster> I certainly understand the pain though :-(
 167 <mars> Ursinha, sound good?  We can review the solution outside the meeting
 168 <gary_poster> or have a hint of it at least
 169 <Ursinha> mars, sure
 170 <mars> thanks gary_poster
 171 <Ursinha> [action] Discuss the solution proposed by gary_poster after the meeting, about ExpatErrors and bug 403606
 172 <ubottu> Launchpad bug 403606 in launchpad-foundations "ExpatError errors should be handled to not generate the OOPSes" [Undecided,New] https://launchpad.net/bugs/403606
 173 <MootBot> ACTION received:  Discuss the solution proposed by gary_poster after the meeting, about ExpatErrors and bug 403606
 174 <Ursinha> thanks mars and gary_poster 
 175 <Ursinha> well
 176 <gary_poster> thanks mars and Ursinha  :-)
 177 <Ursinha> :)
 178 <Ursinha> we're not having more InterfaceErrors (thanks salgado), but we're still having OperationalErrors (OOPS-1306J75, OOPS-1306J96) and DisconnectionErrors (OOPS-1306I440, OOPS-1306J343)
 179 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1306J75
 180 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1306J96
 181 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1306I440
 182 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1306J343
 183 <Ursinha> what can we do about it?
 184 <Ursinha> I think that's yours too mars 
 185 * noodles775_ (n=miken@g225068254.adsl.alicedsl.de) has joined #launchpad-meeting
 186 <mars> Ursinha, I'll need to look at them
 187 <mars> (obviously :)
 188 <Ursinha> :)
 189 <Ursinha> stub, do you have any clues?
 190 <mars> stub, that looks like something for you - Storm barfing while talking over the internal network?
 191 <stub> gary_poster: You could probably switch of the oops in errorlog.py - we already skip certain well defined exceptions entirely.
 192 <gary_poster> stub: oh, ok.  not familiar with that.  sounds good on the face of it.
 193 <mars> stub, gary_poster, well, that's why I suggested wrapping the ExpatError in a 400 status - it's the nice HTTP thing to do, since the client is in the wrong, not us.
 194 <stub> Ursinha: All 'due to administrator command' are when the server killed the connection, usually because it was sitting idle too long.
 195 <gary_poster> mars, can't do that without getting into the publication machinery.  stub's approach is not as elegant as what you suggest, but much more doable in the short term.
 196 <mars> stub, so too much lag between opening the app server request and actually getting to issue commands?
 197 <gary_poster> hm, unless there's an indirection lurking around...checking...
 198 <mars> gary_poster, ok
 199 <stub> mars: For an appserver request, we kill anything that doesn't complete in 2 minutes. For scripts, we kill anything that is idle-in-transaction for 90 minutes (or something like that).
 200 <stub> mars: So a massive slowdown or pause anytime after the db transaction starts
 201 <mars> stub, ok, would you be able to do an analysis, or help me do one, after the meeting?
 202 <mars> to find what is taking so long to execute
 203 <stub> I already looked at this one - I have no idea why that query would stop (the OOPS i'm looking at Ursinha pointed me at earlier)
 204 <mars> you know what the problem is, but I would have to troll the OOPSes to find the root cause
 205 <Ursinha> stub, did I?
 206 <stub> A number of requests timed out all at exactly the same time. I doubt we can reproduce it, and there doesn't seem enough information to diagnose it.
 207 <Ursinha> stub, that was another oops, I guess
 208 <stub> Yup. But the one I'm looking at is select replication_lag() again - this time it took so long it got terminated by the reaper.
 209 <Ursinha> stub, oh, I see
 210 <Ursinha> so they are related
 211 <Ursinha> stub, because of having not enough data in that oops you can't diagnose the Errors?
 212 <Ursinha> *Errors
 213 <Ursinha> mars, stub, we're running out of time, can we discuss that on -dev after the meeting?
 214 <mars> Ursinha, sure
 215 <Ursinha> allright
 216 <Ursinha> [action] mars and stub to discuss the Disconnection and OperationalErrors after the meeting
 217 * noodles775 has quit (Read error: 110 (Connection timed out))
 218 <MootBot> ACTION received:  mars and stub to discuss the Disconnection and OperationalErrors after the meeting
 219 <Ursinha> critical bugs: we have bug 403283, that is in progress as commented on it
 220 <ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
 221 * noodles775_ is now known as noodles775
 222 <Ursinha> moving on!
 223 <Ursinha> thanks a lot guys
 224 <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
 225 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 226 <herb> 2009-07-28 - Rolled critical fixes to the app servers and scripts server.
 227 <herb> We've had some issues with codebrowse in the last week where the process dies but doesn't leave a core file. It's also not leaving anything interesting in the logs. We haven't filed a bug yet, but expect one soon. Any help in determining the best way to debug the issue would be helpful.
 228 <herb> mthaddon has a 2nd librarian instance up and running. We're preparing to load balance between them. mthaddon updated bug #403283 with some questions and it appears stub has responded to them.
 229 <ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
 230 <herb> There aren't any pending queries or cherry pick requests.
 231 <herb> That's it for the LOSAs unless there are questions.
 232 <Ursinha> anything for herb?
 233 <Ursinha> thanks herb!
 234 <Ursinha> moving on
 235 <Ursinha> [TOPIC] * DBA report (stub)
 236 <MootBot> New Topic:  * DBA report (stub)
 237 <herb> thanks Ursinha
 238 <stub> I generated a new database baseline from the production database and landed it. We do this occasionally to ensure that the version of the db we are developing on matches what is actually running on production (patches made to the live system not backported to the trunk or stuffups can cause drift).
 239 <stub> If you have an approved but unlanded database patch you will need a new database patch number from me.
 240 <Ursinha> stub, oot?
 241 <Ursinha> :)
 242 <stub> oot
 243 <Ursinha> sweet
 244 <Ursinha> anyone wants to say something?
 245 <Ursinha> thanks stub :)
 246 <Ursinha> I have something to say to all contacts
 247 <Ursinha> we want to close the 2.2.7 milestone, so, if you have pending bugs or blueprints, please, retarget of fix release them 
 248 <Ursinha> intellectronica, I saw that bugs team has a lot (really) of not assigned bugs targeted to 2.2.7
 249 <intellectronica> Ursinha: sure, will make sure we sort them out now
 250 * jtv has quit (Read error: 60 (Operation timed out))
 251 <Ursinha> thanks a lot intellectronica 
 252 <Ursinha> and all guys
 253 <Ursinha> you rock
 254 <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
 255 <Ursinha> #endmeeting
 256 <MootBot> Meeting finished at 10:47.