1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <gary_poster> me
   8 <mrjazzcat> me
   9 <Ursinha> me me
  10 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
  11 <matsubara> sinzui, Chex, rockstar, bigjools, allenap, danilos: hi
  12 <bigjools> \o
  13 <allenap> me
  14 <rockstar> hi
  15 <sinzui> me
  16 <Chex> hello
  17 <danilos> me
  18 <matsubara> ok, everyone is here.
  19 <matsubara> [TOPIC] Agenda 
  20 <MootBot> New Topic:  Agenda
  21 <matsubara>  * Actions from last meeting
  22 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  23 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  24 <matsubara>  * DBA report (stub)
  25 <matsubara>  * Proposed items
  26 <matsubara> [TOPIC] * Actions from last meeting
  27 <MootBot> New Topic:  * Actions from last meeting
  28 <matsubara>  * matsubara to discuss with foundations bug 403281 and set importance on it
  29 <matsubara>         * discussed with Gary and it's now triaged and an importance set.
  30 <matsubara>     * rockstar to triage bug 531687
  31 <matsubara>     * matsubara to discuss bug 531965 with foundations as well
  32 <matsubara>         * discussed with Gary and it's now triaged and an importance set.
  33 <matsubara>     * rockstar to investigate failures on mpcreationjobs and sendbranchmail scripts
  34 <matsubara>         * <rockstar> matsubara, abentley is already figuring out what's going on with those scripts.
  35 <matsubara>     * rockstar to investigate Out-Of-Memory error: https://pastebin.canonical.com/28737/ and get back to losas about it
  36 <matsubara>     * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
  37 * flacoste (~francis@canonical/launchpad/flacoste) has joined #launchpad-meeting
  38 <ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Low,Fix committed] https://launchpad.net/bugs/403281
  39 <ubottu> Launchpad bug 531687 in launchpad-code "Accessing a merge proposal during the rollout (ie R/O mode) oopsed" [High,Triaged] https://launchpad.net/bugs/531687
  40 <ubottu> Launchpad bug 531965 in launchpad-foundations "During the rollout, database changes caused old code to oops" [Low,Invalid] https://launchpad.net/bugs/531965
  41 <matsubara> I didn't add the count of untriaged bugs per team to the oops section yet
  42 <matsubara> [action] * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
  43 <MootBot> ACTION received:  * matsubara and/or Ursinha to add a count of untriaged bugs per team to the oops section
  44 <Ursinha> yes sir
  45 <matsubara> rockstar, any news about the OOM error? did you get back to losas about it?
  46 <matsubara> I see that 531687 was properly triaged and an importance set. thanks for that rockstar 
  47 <rockstar> matsubara, others on my team are looking into it.  We think it's probably just another scalability issue.
  48 <matsubara> cool. thanks rockstar 
  49 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  50 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  51 <matsubara> I have 3 bugs for today, 2 of them were already dealt with
  52 <matsubara> bug 535071 still need some foundations love
  53 <ubottu> Launchpad bug 535071 in launchpad-foundations "Better error handling when librarian is offline" [Undecided,New] https://launchpad.net/bugs/535071
  54 <matsubara> gary_poster, can you take a look ^? 
  55 <matsubara> fwiw, the other two are bugs 535078, 535672
  56 <ubottu> Launchpad bug 535078 in launchpad-foundations "InvalidURIError raised accessing an API url" [High,Triaged] https://launchpad.net/bugs/535078
  57 <ubottu> Launchpad bug 535672 in malone "TransactionRollbackError using the api to find users affected with dupes" [Undecided,Won't fix] https://launchpad.net/bugs/535672
  58 <gary_poster> looking (how did I miss this? :-/ )
  59 <matsubara> thanks gary_poster 
  60 <matsubara> we have 4 critical bugs, 3 fix committed and 1 in progress.
  61 <bigjools>  /o\
  62 <matsubara> for the failing scripts
  63 <matsubara> we have two failing frequently: productreleasefinder and oops-pruner
  64 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=pruner
  65 <gary_poster> heh
  66 <Ursinha> ubottu: there, there
  67 <matsubara> I replied to one of the oops pruner failure email but didn't get a reply yet.
  68 <matsubara> Chex, could you take a look in the oops pruner script logs and check why it's failing?
  69 <Chex> matsubara: yes sure, I can look at that
  70 <matsubara> Chex, please reply to one of the failures emails with the traceback or file a bug
  71 <matsubara> [action] Chex to verify why oops-pruner is failing and reply to the failure email with more information.
  72 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=pruner
  73 <MootBot> ACTION received:  Chex to verify why oops-pruner is failing and reply to the failure email with more information.
  74 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=pruner
  75 <matsubara> sinzui, how about the prf script?
  76 <matsubara> been failing on a daily basis for the past week
  77 <sinzui> It is always running 2 hours late
  78 <sinzui> It always runs, just late
  79 <sinzui> I think we need to consider changing the threshold, or changing the order
  80 <matsubara> sinzui, so we should adjust the monitoring script to take that into account, shouldn't we?
  81 <matsubara> [action] sinzui to work with losas to set the increase threshold for the product release finder monitoring script.
  82 <MootBot> ACTION received:  sinzui to work with losas to set the increase threshold for the product release finder monitoring script.
  83 <matsubara> I think that's all for today's oops/scripts/critical section. thanks everyone
  84 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
  85 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
  86 <Chex> hello everyone, here is this weeks report:
  87 <Chex> - We are in the process of working on migrating to using SSO run by ISD branded for LP (open-ID single signon) in
  88 <Chex>         Launchpad.
  89 <Chex> - The LP DBs on one of the slave DB servers may need to be rebuilt/repacked as we're getting short on
  90 <Chex>         space on that server.
  91 <Chex> - LP incidents of note:
  92 <Chex>         ; LP Cherry-picks: CP to crowberry/codehost; revno 9082
  93 <Chex>         ; There was a buildbot error that stopped merging from db-devel to db-stable for a few days, which has
  94 <Chex>                 been fixed.
  95 <Chex> and thats all for us.  Any questions/comments??
  96 <matsubara> Chex, I take you're working with stub to have the db rebuilding done?
  97 <Chex> oh, and the buildbot merge error seems to be happening again, so this still needs to be investigated.
  98 <Chex> matsubara: yes I believe we will work with stub on the db rebuild.
  99 <matsubara> rockstar, can you take that? that's a build engineer thing, right?
 100 <rockstar> matsubara, I'm not build engineer anymore.
 101 <matsubara> rockstar, oh, who's the build engineer for this cycle?
 102 <rockstar> matsubara, I don't think there is one.
 103 <matsubara> I guess it'll fallback to foundations then. gary_poster can you work with losas re: the buildbot issue above?
 104 <gary_poster> matsubara: yes
 105 <matsubara> cool. thanks.
 106 <matsubara> [TOPIC] * DBA report (stub)
 107 <MootBot> New Topic:  * DBA report (stub)
 108 <matsubara> sorry, forgot to thank Chex for his report. thank you Chex!
 109 <matsubara> dba report was sent to the list. any question please follow up there
 110 <matsubara> [TOPIC] * Proposed items
 111 <MootBot> New Topic:  * Proposed items
 112 <matsubara> no new proposed items
 113 <matsubara> and I guess that's all for today folks. anything else before I close?
 114 <Chex> matsubara: no problem!
 115 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 116 <matsubara> #endmeeting 
 117 <MootBot> Meeting finished at 10:23.