1 <Ursinha> #startmeeting
   2 <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   3 <Ursinha> [TOPIC] Roll Call
   4 <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
   5 <MootBot> Meeting started at 10:02. The chair is Ursinha.
   6 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   7 <MootBot> New Topic:  Roll Call
   8 <bigjools> me
   9 <Ursinha> me
  10 <henninge> ich
  11 <sinzui> me
  12 <rockstar> me
  13 * flacoste (n=francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
  14 <flacoste> me
  15 <herb> me
  16 <Ursinha> herb, intellectronica, hi
  17 <Ursinha> :)
  18 <Ursinha> who's missing?
  19 <Ursinha> stub is missing, but he can join later
  20 * mthaddon (n=mthaddon@adsl-70-137-141-190.dsl.snfc21.sbcglobal.net) has joined #launchpad-meeting
  21 <Ursinha> intellectronica is missing too
  22 <Ursinha> let's move on
  23 <Ursinha> [TOPIC] Agenda
  24 <Ursinha> * Actions from last meeting
  25 <Ursinha> * Oops report & Critical Bugs
  26 <Ursinha> * Operations report (mthaddon/herb/spm)
  27 <Ursinha> * DBA report (stub)
  28 <MootBot> New Topic:  Agenda
  29 <Ursinha> [TOPIC] * Actions from last meeting
  30 <Ursinha> * Ursinha to talk to intellectronica about bug 357316
  31 <Ursinha> * Ursinha to talk to henninge about bug 302449
  32 <Ursinha> * rockstar to confirm that bzr fix for bug 360791 was applied to LP's bzr tree.
  33 <Ursinha> * cprov to request CP of fix for bug 370513
  34 <MootBot> New Topic:  * Actions from last meeting
  35 <ubottu> Launchpad bug 357316 in malone "hwdb +submit failing with KeyError OOPS" [Undecided,Triaged] https://launchpad.net/bugs/357316
  36 <ubottu> Launchpad bug 302449 in rosetta "Uploading a file with the same name triggers a database constraint." [Medium,Triaged] https://launchpad.net/bugs/302449
  37 <Ursinha> I suck and failed mine
  38 <ubottu> Launchpad bug 360791 in bzr/1.14 "bzr pull/branch shows "Error received from smart server: ('NoSuchRevision',)"" [Critical,In progress] https://launchpad.net/bugs/360791
  39 <ubottu> Launchpad bug 370513 in soyuz "failure to accept PPA uploads" [Critical,Fix committed] https://launchpad.net/bugs/370513
  40 <Ursinha> [action] Ursinha to talk to intellectronica about bug 357316
  41 <MootBot> ACTION received:  Ursinha to talk to intellectronica about bug 357316
  42 <Ursinha> henninge, hi :)
  43 <henninge> I think danilo was onto that ...
  44 <rockstar> I don't know if the fix has been cherry picked into production...
  45 <Ursinha> henninge, can you just confirm that, please? it's set as medium, do we use that status?
  46 <henninge> Ursinha: in rosetta we do ;)
  47 <Ursinha> rockstar, hm, can you check that too?
  48 <rockstar> Code team does, for things of medium importance.
  49 <bigjools> as does Soyuz
  50 <herb> rockstar: it was cherry picked on 2009-05-09
  51 <Ursinha> I rarely see medium statuses :) that's why I'm asking
  52 <Ursinha> thanks herb
  53 <rockstar> herb: cool, thanks.
  54 <Ursinha> [action] henninge to check with danilo the status of bug 302449
  55 <MootBot> ACTION received:  henninge to check with danilo the status of bug 302449
  56 <ubottu> Launchpad bug 302449 in rosetta "Uploading a file with the same name triggers a database constraint." [Medium,Triaged] https://launchpad.net/bugs/302449
  57 <Ursinha> cool
  58 <Ursinha> moving on then
  59 <Ursinha> [TOPIC] * Oops report & Critical Bugs
  60 <Ursinha> there's only one worth mentioning, that is the one causing the InterfaceError oopses, we're still having lots and lots of occurrences (bugs 374909 and 376207), seems to be worked on by jamesh, is that correct flacoste?
  61 <MootBot> New Topic:  * Oops report & Critical Bugs
  62 <ubottu> Launchpad bug 374909 in storm "InterfaceError: connection already closed should be converted into DisconnectionError" [High,Triaged] https://launchpad.net/bugs/374909
  63 <ubottu> Launchpad bug 376207 in launchpad-foundations "LaunchpadOpenIDStore doesn't support database disconnection" [High,In progress] https://launchpad.net/bugs/376207
  64 <flacoste> so, jamesh is working on 374909
  65 <flacoste> and the other one also
  66 <Ursinha> right
  67 <flacoste> but stuart has an easy fix for the later, that I'll likely asked to be cherrypicked
  68 <flacoste> we can deploy jamesh proper fix during next roll-out
  69 <Ursinha> flacoste, hm, good
  70 <Ursinha> flacoste, about the cp, when do you think it can be done?
  71 <flacoste> i didn't look at the branch
  72 <herb> 374909 is still cropping up from time to time, btw. though it's much gooder(tm) than it was last week.
  73 <flacoste> but once i approved it, as soon as the LOSA can take care of it
  74 <Ursinha> flacoste, right. okay
  75 <flacoste> i don't think i'll ask a C-P of the INterfaceError
  76 <flacoste> (storm fix being worked on by jamesh)
  77 <herb> flacoste: why?
  78 <flacoste> well, because that's not a root-cause
  79 <flacoste> so with the other fix in place, we shouldn't see it
  80 <herb> ok
  81 <flacoste> that fix is more prophylactic
  82 <Ursinha> flacoste, who's investigating the root cause?
  83 <flacoste> if you are talking about why we are getting disconnection errors in the first place
  84 <flacoste> nobody, really
  85 <flacoste> but we have a fix for the places where we should be trapping those
  86 <flacoste> and recovering
  87 <Ursinha> yes, the disconnection errors
  88 <flacoste> we have no ideas at why it's happening
  89 <flacoste> there is nothing in the DB logs
  90 <flacoste> about them
  91 <Ursinha> this is creepy
  92 <Ursinha> the fix you say you have
  93 <Ursinha> is inside the fixes for one of those bugs?
  94 <flacoste> yes
  95 <Ursinha> great
  96 <Ursinha> so, you'll ask a cp for the second bug
  97 <flacoste> exactly
  98 <Ursinha> [action] flacoste to ask a cp for fix for bug 376207 after reviewing it
  99 <MootBot> ACTION received:  flacoste to ask a cp for fix for bug 376207 after reviewing it
 100 <ubottu> Launchpad bug 376207 in launchpad-foundations "LaunchpadOpenIDStore doesn't support database disconnection" [High,In progress] https://launchpad.net/bugs/376207
 101 <Ursinha> awesome
 102 <Ursinha> we have one critical bug, being worked on
 103 <Ursinha> so, unless anyone has anything to point, moving to the next section!
 104 <Ursinha> good
 105 <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
 106 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 107 <herb> 2009-05-07 - Cherry pick r8906 to the scripts server and r122 of storm to lpnet* & edge*
 108 <herb> 2009-05-09 - Cherry pick r4006 of bzr to the codehosting server and r123 of storm to lpnet* & edge*
 109 <herb> 2009-05-09 - Cherry pick r8348, r8312 to the PPA server and r8376 to lpnet*
 110 <herb> 2009-05-10 - mailman didn't have access the necessary access to the DB server, but it was only noticed after restarting for log rotation. mailing lists were unavailable for approximately 7 hours.
 111 <herb> We still seem to be encountering bug #156453 and bug #118626, but the situation is much improved since the rollout.
 112 <herb> flacoste: cprov requested a rollout of the current production tree to cesium. Apparently there was a critical fix that was included in while in RC, but we didn't re-roll to cesium. Can you (dis)approve that?
 113 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
 114 <ubottu> Launchpad bug 118626 in bzr-email "plugin documentation does not make interaction with checkouts clear" [Medium,Confirmed] https://launchpad.net/bugs/118626
 115 <bigjools> herb: it's already approved by kiko
 116 <flacoste> herb: i think kiko did? but otherwise, i can look into it
 117 <flacoste> right
 118 <herb> bigjools: missed it.  thanks
 119 <Ursinha> any other questions to herb?
 120 <Ursinha> okay
 121 <Ursinha> [TOPIC] * DBA report (stub)
 122 <MootBot> New Topic:  * DBA report (stub)
 123 <rockstar> herb: I was under the impression that the loggerhead stuff was WAY better than "much improved"
 124 <flacoste> stub sent it to the list
 125 <herb> rockstar: we're still restarting a couple of times a week. which is much improved over a couple of times a day.
 126 <herb> rockstar: order of magnitude.
 127 <Ursinha> flacoste, to lp list? I can't seem to find it
 128 <rockstar> herb: okay.  Is it memory restarts, or hanging restarts
 129 <flacoste> The ex-master database (launchpad_prod on hackberry) is bloated and
 130 <flacoste> started exceeding its free space map settings. Nothing really to worry
 131 <flacoste> about - it might cause bloat to spiral but I suspect not in this case.
 132 <flacoste> The losas can bounce it after shutting down the systems using it as a
 133 <flacoste> slave, and I've suggested using it as the standalone replica for
 134 <flacoste> read-only mode launchpad during the rollout because we then rebuild it
 135 <rockstar> (The memory restarts shouldn't be happening anymore)
 136 <flacoste> afterwards and it will be all nice and freshly packed.
 137 <flacoste> Nothing major do do with database patches this cycle. rockstar's
 138 <flacoste> bugbranch and specbranch column pruning needs to be cleared with Mark
 139 <flacoste> still.
 140 <Ursinha> thanks flacoste 
 141 <rockstar> flacoste: yeah, and there are other branches dependent on that one.
 142 <herb> rockstar: The memory situation seems ok (ie. not death spiraling).  seems to be hangs at this point.
 143 <herb> rockstar: we still see it ~1.5GB resident, but doesn't seem to grow beyond that.
 144 <rockstar> herb: well, "death spiraling" to be is different than the memory issues.
 145 <rockstar> herb: it's going to be *kinda* memory intensive just because of what it's serving.
 146 <herb> rockstar: understood
 147 <herb> rockstar: as I said, much improved. 1.2 - 1.5G is much better than 3.7G
 148 <rockstar> herb: and we know the cause of the hangs, we just don't know how to fix it.
 149 <herb> rockstar: good news, bad news, eh?
 150 <rockstar> herb: yeah, something like that.
 151 <Ursinha> okay. anyone else want to say something?
 152 <Ursinha> 5
 153 <Ursinha> 4
 154 <Ursinha> 3
 155 <Ursinha> 2
 156 <Ursinha> 1
 157 <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
 158 <Ursinha> #endmeeting