1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 09:04. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <sinzui> me
   6 <matsubara> [TOPIC] Roll Call 
   7 <MootBot> New Topic:  Roll Call
   8 <allenap> me
   9 <matsubara> sorry for being late, I was on my stand up meeting
  10 <matsubara> stub, Chex, gary_poster, rockstar, bigjools, danilo_: Hi
  11 <gary_poster> matsubara: hi
  12 <danilos> me
  13 <bigjools> matsubara: TLs are in a call.... :/
  14 <bigjools> noodles775: can you cover me please?
  15 <matsubara> bigjools, right, by the end of this meeting sinzui will propose a change for the production meeting so it won't clash anymore
  16 <noodles775> bigjools: OK.
  17 <bigjools> ta
  18 <danilos> bigjools, we've got an agenda item to move the call, so you should be in for at least that
  19 <matsubara> ok, so let's move on. Chex, rockstar and stub can join in later
  20 <matsubara> [TOPIC] Agenda
  21 <matsubara>  * Actions from last meeting
  22 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  23 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  24 <matsubara>  * DBA report (stub)
  25 <matsubara>  * Proposed items
  26 <MootBot> New Topic:  Agenda
  27 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  28 <matsubara> [TOPIC] * Actions from last meeting
  29 <MootBot> New Topic:  * Actions from last meeting
  30 <matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
  31 <matsubara> * matsubara to chase someone from code team about bug 480000
  32 <matsubara> * matsubara to chase code people about code script failures (create-merge-proposals, branch puller and update branches)
  33 <matsubara> * matsubara to ask someone from code about bug 485318
  34 <matsubara>     * emailed tim about these
  35 <matsubara> * Chex to follow up with thumper about the multiple git import failures on the importd
  36 <matsubara> * matsubara to file a bug for OOPS-1420ED1047
  37 <matsubara>     * there was a bug filed for this already. Bug 484368
  38 <matsubara> * sinzui to investigate failure on the mirror prober (The script 'distributionmirror-prober' didn't run on 'loganberry' between 2009-11-23 06:07:04 and 2009-11-23 12:07:04 (last seen 2009-11-23 04:54:10.444057))
  39 <matsubara> * matsubara to ask gary about python2.5 update and get back to losas
  40 <matsubara>     * francis emailed the list and gary about this
  41 <matsubara> * matsubara to ask stub to contact losas about load increase on wildcherry
  42 <matsubara>     * emailed stub about this one
  43 <ubottu> Launchpad bug 480000 in launchpad-code "OOPS deleting a branch" [Low,Triaged] https://launchpad.net/bugs/480000
  44 <ubottu> Launchpad bug 485318 in launchpad-code "POSTToNonCanonicalURL error using bazaar client" [Wishlist,Triaged] https://launchpad.net/bugs/485318
  45 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1420ED1047
  46 <ubottu> Launchpad bug 484368 in rosetta "LocationError: 'top_projects_and_packages_to_translate'" [High,Triaged] https://launchpad.net/bugs/484368
  47 <matsubara> I don't recall an email explaining the qa-tags experiment but Ursula did show up a wiki page for me before leaving on vacation
  48 <sinzui> matsubara: the script was dealyed, ran fine later. The same thing happened to the PRF. it is running fine
  49 <matsubara> danilo_, do you know if that email was done?
  50 <matsubara> sinzui, thanks for checking
  51 <matsubara> s/done/sent/
  52 * salgado is now known as salgado-lunch
  53 <matsubara> mthaddon, around?
  54 <matsubara> I guess people are too busy with other stuff
  55 <matsubara> let's move on
  56 <matsubara> [action] * Ursinha to send one email to lp list explaining the qa-tags experiment
  57 <MootBot> ACTION received:  * Ursinha to send one email to lp list explaining the qa-tags experiment
  58 <matsubara> [action] * Chex to follow up with thumper about the multiple git import failures on the importd
  59 <MootBot> ACTION received:  * Chex to follow up with thumper about the multiple git import failures on the importd
  60 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  61 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  62 <matsubara> danilo_,  is https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1425EA795 and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1429EB593 related to bug https://bugs.edge.launchpad.net/rosetta/+bug/484368?
  63 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1425EA795
  64 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1429EB593
  65 <ubottu> Ubuntu bug 484368 in rosetta "LocationError: 'top_projects_and_packages_to_translate'" [High,Triaged]
  66 <matsubara> sinzui, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1430F2574
  67 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
  68 <matsubara> sinzui, is this registry or foundations?
  69 <sinzui> foundations
  70 <sinzui> matsubara: high/critical
  71 <sinzui> matsubara: This may be caused by the replication delay
  72 <matsubara> all right. I'll file a bug for it and ask gary_poster to take a look
  73 <matsubara> [action] matsubara to file a high/critical bug for OOPS-1430F2574
  74 <MootBot> ACTION received:  matsubara to file a high/critical bug for OOPS-1430F2574
  75 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
  76 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
  77 <matsubara> rockstar, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1427EA45 I've seen 4 occurrences of this oops last week, is this a known issue? some bad data? worth a bug?
  78 <matsubara> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536 shouldn't this one be a NotFound rather than a NotFoundError? 
  79 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1427EA45
  80 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
  81 <matsubara> I guess I'll have to email tim about those as well
  82 <gary_poster> OOPS-1430F2574 : I agree that this is probably replication
  83 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
  84 <matsubara> [action] matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1427EA45 I've seen 4 occurrences of this oops last week, is this a known issue? some bad data? worth a bug?
  85 <matsubara> <matsubara> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536 shouldn't this one be a NotFound rather than a NotFoundError? 
  86 <MootBot> ACTION received:  matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1427EA45 I've seen 4 occurrences of this oops last week, is this a known issue? some bad data? worth a bug?
  87 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1427EA45
  88 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
  89 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1427EA45
  90 <matsubara> damn
  91 <matsubara> [action] matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536  shouldn't this one be a NotFound rather than a NotFoundError?
  92 <MootBot> ACTION received:  matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536  shouldn't this one be a NotFound rather than a NotFoundError?
  93 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
  94 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
  95 <matsubara> we have 5 critical bugs, 4 of them fix committed and 1 in progress
  96 <matsubara> so all good in that area
  97 <matsubara> no script failures since last week (well, only PRF but that's fine per sinzui)
  98 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
  99 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 100 <matsubara> let's move to the next topic as there's no losa around
 101 <matsubara> [TOPIC] * DBA report (stub)
 102 <MootBot> New Topic:  * DBA report (stub)
 103 <stub> We have had two incidents where appserver requests have sent the load on the main database server over 100 in some sort of a feedback loop we dubbed the DB Death Spiral. We think we tracked down the trigger - the page the load balancers used to detect if Launchpad is up accessed the session database, and our session machinery becomes a bottleneck under load.
 104 <stub> What we hope is the immediate fix lands tomorrow - stopping that page from accessing the database. I have plans of offloading the bulk of the session machinery work to memcache so it should stop becoming a bottleneck under load, but that is work for the next cycle or two.
 105 <stub> We also managed to have replication issues, because when it rains it pours. Both times where do do with adding a new replica into the cluster.
 106 <stub> The first time, it turned out some events where left around that should have been cleared up causing conflicts. So when one of our replicas tried to confirm it had seen an event, it found the confirmation was already there so it aborted.
 107 <stub> The second one, today, removing the replica from the cluster hadn't quite succeeded so replication lag on the cluster was increasing. This wasn't noticed or was ignored, and we attempted to re-add the database back into a heavily lagged cluster. This needed recovering. I don't think users where affected today.
 108 * danilo_ has quit (Read error: 110 (Connection timed out))
 109 <stub> And that is all I've typed so far ;)
 110 <matsubara> stub, should I expect to see lots of OOPSes in the reports about this replication lag issue?
 111 <stub> I've got a bug open to add some more safety belts to our helpers to catch these cases.
 112 <stub> matsubara: Hopefully not. I'm not sure though.
 113 <matsubara> stub, all right. I'll let you know if spot anything 
 114 <matsubara> thanks stub
 115 <matsubara> [TOPIC] * Proposed items
 116 <MootBot> New Topic:  * Proposed items
 117 <matsubara> # Move the production meeting one hour later to avoid clash with other meetings (sinzui) 
 118 <sinzui> please
 119 <sinzui> I am in another meeting right now
 120 <matsubara> I'm +1 on the change. it'd be actually better for me to have the meeting at 16UTC
 121 <matsubara> how about the others?
 122 <noodles775> I'm assuming that bigjools is also +1 for the same reason.
 123 <matsubara> and danilo too
 124 <bigjools> +1
 125 <sinzui> +1
 126 <stub> That is getting nuts for me, but I can do the report by email just as easily as typing it up here.
 127 <matsubara> on the other hand, what do you think about not having this meeting at all anymore? do you think it's useful or the format could be changed? I see lots of people missing this meeting or not paying much attention...
 128 <matsubara> stub, your section and the losas section are the ones that interest me the most :-)
 129 <matsubara> stub, reports by email are fine by me, not sure about others
 130 <stub> I tend to think email would be a better forum rather than playing Chinese whispers.
 131 <matsubara> yeah
 132 <matsubara> [action] matsubara to talk to TL about not having the LP production meeting anymore or change its format
 133 <MootBot> ACTION received:  matsubara to talk to TL about not having the LP production meeting anymore or change its format
 134 <matsubara> and for the next one, let's try to have it at 16UTC. I'll email the QA contacts to let everyone know.
 135 <matsubara> [action] matsubara to email Qa contacts about next LP prod. meeting at 16UTC
 136 <MootBot> ACTION received:  matsubara to email Qa contacts about next LP prod. meeting at 16UTC
 137 <stub> At that hour, it will be drunk from a gogo bar :)
 138 <matsubara> [action] matsubara to email losas about their weekly report
 139 <MootBot> ACTION received:  matsubara to email losas about their weekly report
 140 <matsubara> hehe
 141 <matsubara> and I think that's all for today
 142 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 143 <noodles775> Thanks matsubara
 144 <matsubara> #endmeeting 
 145 <MootBot> Meeting finished at 09:34.