1 <matsubara> #startmeeting
   2 <bigjools> me
   3 <MootBot> Meeting started at 11:00. The chair is matsubara.
   4 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   5 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   6 <matsubara> [TOPIC] Roll Call 
   7 <MootBot> New Topic:  Roll Call
   8 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   9 <matsubara> chex, gary_poster, rockstar, bigjools, allenap: hi
  10 <gary_poster> me, sorry
  11 * rockstar are here
  12 <bigjools> I meed earlier
  13 <matsubara> sorry bigjools, missed that
  14 <matsubara> apologies from Translations. they're sprinting
  15 <bigjools> easily missed I was premature :)
  16 <chex> here
  17 <allenap> me
  18 * sinzui (~sinzui@pool-71-163-158-16.washdc.fios.verizon.net) has joined #launchpad-meeting
  19 <matsubara> hi sinzui 
  20 <sinzui> me
  21 <matsubara> ok, everyone is here
  22 <matsubara> [TOPIC] Agenda 
  23 <MootBot> New Topic:  Agenda
  24 <matsubara>  * Actions from last meeting
  25 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  26 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  27 <matsubara>  * DBA report (stub)
  28 <matsubara>  * Proposed items
  29 <matsubara> [TOPIC] * Actions from last meeting
  30 <MootBot> New Topic:  * Actions from last meeting
  31 <matsubara>  * Ursinha to send email to stub about the dba report
  32 <matsubara>  * sinzui to follow up in work with losas to set the increase threshold for the product release finder monitoring script
  33 <matsubara>  * matsubara to talk to gary_poster about bug 535071
  34 <matsubara>    * Done.
  35 <matsubara>  * Ursinha to send email to TLs about QA summaries
  36 <matsubara>  * matsubara to triage open oops-tools bugs
  37 <matsubara>  * Ursinha to triage open lp-qa-tools bugs
  38 <ubottu> Launchpad bug 535071 in launchpad-foundations "Better error handling when librarian is offline" [High,Triaged] https://launchpad.net/bugs/535071
  39 * danilos (~danilo@canonical/launchpad/danilos) has joined #launchpad-meeting
  40 <matsubara> I did some triage for oops-tools bugs, still some to look at
  41 <Ursinha> matsubara: done the 1st and the 3rd, not the 2nd
  42 <sinzui> matsubara: I have instructions from spm regarding how to change the window or schedule. I will try to land a change next week
  43 <matsubara> [action] * Ursinha to send email to TLs about QA summaries
  44 <MootBot> ACTION received:  * Ursinha to send email to TLs about QA summaries
  45 <matsubara> [action]  * matsubara to triage open oops-tools bugs
  46 <MootBot> ACTION received:   * matsubara to triage open oops-tools bugs
  47 <matsubara> thanks sinzui and Ursinha 
  48 <Ursinha> np matsubara
  49 <matsubara> sinzui, should I add the action item again or can I consider it as done?
  50 <sinzui> matsubara: yes. pease do
  51 <matsubara> [action]  * sinzui to follow up in work with losas to set the increase threshold for the product release finder monitoring script
  52 <MootBot> ACTION received:   * sinzui to follow up in work with losas to set the increase threshold for the product release finder monitoring script
  53 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  54 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  55 <matsubara> We're still seeing https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1541O1653 which seems to be a regression of bug 527985
  56 <ubottu> Launchpad bug 527985 in launchpad-foundations "backing out of openid dance creates oops" [High,Fix released] https://launchpad.net/bugs/527985
  57 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1541O1653
  58 <matsubara> another one that seems to be a regression: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1545E465 --> bug 408738
  59 <ubottu> Launchpad bug 408738 in malone "OOPS when rendering bug activity" [High,Fix released] https://launchpad.net/bugs/408738
  60 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
  61 <matsubara> and https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1544A949 regression of bug 508302
  62 <ubottu> Launchpad bug 508302 in malone "NotImplementedError OOPS when reporting a bug" [High,Fix released] https://launchpad.net/bugs/508302
  63 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
  64 <matsubara> sinzui, timeout retrieving vouchers: https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1543EA744, I remember a bug about that but couldn't find it
  65 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1543EA744
  66 <matsubara> not sure if it's the same thing
  67 <sinzui> matsubara: that was brad and myself
  68 <sinzui> matsubara: ISD is still making changes to services we depend on
  69 <matsubara> gary_poster, can you look at foundations one ^?
  70 <sinzui> matsubara: at this time we are fortunate that only bac and myself are affected
  71 <gary_poster> looking
  72 <matsubara> allenap, can you take a look at the two bugs above? 
  73 * allenap looks
  74 <bac> sinzui: argh
  75 <matsubara> sinzui, so should we just ignore those oopses for now? is there a bug that we could link to that oops report?
  76 * sinzui read the oopses and knows what bac is up to during his work hours
  77 <sinzui> matsubara: I have not had a problem in the last two days.
  78 <sinzui> matsubara: If it persists I will report a bug on the correct ISD project,
  79 <bac> sinzui, matsubara : i see that user's project has had a voucher applied so perhaps it was spurious.  have we heard from him?
  80 <matsubara> bac, nope, I caught that in the oops summary
  81 <sinzui> matsubara: bac: this was a networking issue with saleforce. The reason you could not find the bug is that ISD retargeted the other bugs to their porjects
  82 <allenap> matsubara: I'll file a new bug for OOPS-1545E465; the linked bug is not right.
  83 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
  84 <matsubara> sinzui, looks like the oops is triggerable by regular users as well
  85 <sinzui> matsubara: anyone using +vouchers is a regular user
  86 <allenap> matsubara: I'll talk to deryck about OOPS-1544A949.
  87 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
  88 <matsubara> [action] allenap to discuss with deryck about OOPS-1544A949 and file a bug for OOPS-1545E465.
  89 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
  90 <MootBot> ACTION received:  allenap to discuss with deryck about OOPS-1544A949 and file a bug for OOPS-1545E465.
  91 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
  92 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1544A949
  93 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
  94 <sinzui> matsubara:  the oops is not in launchpad, it is salesforce. We started talking to ISD last month when vouchers broke
  95 <matsubara> [action] matsubara to fix linked bug for OOPS-1545E465
  96 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
  97 <MootBot> ACTION received:  matsubara to fix linked bug for OOPS-1545E465
  98 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1545E465
  99 <gary_poster> matsubara: I'll be discussing my oops with salgado and filing a new bug or something
 100 <matsubara> allenap, gary_poster, thanks
 101 <gary_poster> (s/or something/or reopening the old one)
 102 <matsubara> [action] gary_poster to discuss OOPS-1541O1653 with salgado and file a bug for it.
 103 <MootBot> ACTION received:  gary_poster to discuss OOPS-1541O1653 with salgado and file a bug for it.
 104 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1541O1653
 105 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1541O1653
 106 <matsubara> sinzui, in any case, lp should degrade gracefully when salesforce is offline or timing out
 107 <sinzui> DUDE! I know that
 108 <sinzui> This is not the channel to discuss ISD's changes to systems that affect launchpad and landscape
 109 <matsubara> ok lebowski, I know that you know that. just making sure we're in the same page. I'm more interested in the changes that we can make to avoid the oops from happening (i.e. degrade gracefully)
 110 <gary_poster> lol
 111 <sinzui> I favor removing salesforce. It is not my decision. We can evaluate our dependencies on outsiders when we start NewCo
 112 <matsubara> sinzui, rather than removing salesforce, can't we show a page saying that LP couldn't talk to salesforce while the duderino was browsing +vouchers? much like as your proposed solution to bug 267852?
 113 <ubottu> Launchpad bug 267852 in launchpad-foundations "google search timeouts are not usable" [Low,Triaged] https://launchpad.net/bugs/267852
 114 <sinzui> matsubara: because it is an oops. there is nothing to show nor any action to take. I could not license Canonical projects for an hour
 115 <bigjools> sigh - s/note/not/
 116 <bigjools> oops
 117 <matsubara> sinzui, makes sense then. I'll let you know if I see more of these OOPS and then you can escalate with ISD. thanks sinzui 
 118 <matsubara> we have 5 critical bugs
 119 <matsubara> 3 fix committed, one in progress and one confirmed
 120 <matsubara> the one confirmed is assigned to me and I'll take care of it today
 121 <matsubara> the only broken script is flag-expired-memberships, which stub will take care of
 122 <matsubara> thanks everyone, let's move on
 123 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 124 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 125 * matsubara hands the mic to chex 
 126 <chex> hello everyone
 127 <chex> here is the LOSA report for this week:
 128 <chex> - New ImportD server 'pear' has been setup this week and is processing jobs now.
 129 <chex> - PQM is closing at 2300 UTC Friday for LP roll-out next week.
 130 <chex> - SSO DB Migration project away from LP DB is delayed due to some issues, does anyone know
 131 <chex>         where we are placed with this now?
 132 <chex> - LP incidents of note: ; LP Cherry-picks:
 133 <chex>         22-Mar: CP 9095 to lpnet*, & codehost
 134 <chex>         24-Mar: CP 9097 to loganberry/checkwatches and 9098 to shipit*
 135 <chex> and thats the report for this week, any comments/questions??
 136 <matsubara> gary_poster, do you know about the SSO DB migration ^?
 137 <gary_poster> chex, matsubara, yes
 138 <gary_poster> chex, losas were cc'd on relevant emails
 139 <gary_poster> summary: um...
 140 <gary_poster> other Canonical openid apps that relied on launchpad team membership for privileges were going to break.
 141 <gary_poster> We have to handle that.
 142 <gary_poster> A plan is now in place to do so
 143 <gary_poster> It will involve some DB changes that stub is working on, and some changes for ISD
 144 <chex> gary_poster: ok, great, sounds good then, thanks
 145 <gary_poster> We regard splitting the by no later than two weeks from now as an absolute necessity, because of various pressures such as HD space and other excitement, and hope to have it addressed sooner (beginning of week after next)
 146 <matsubara> thanks gary_poster, chex 
 147 <matsubara> [TOPIC] * DBA report (stub)
 148 <MootBot> New Topic:  * DBA report (stub)
 149 <matsubara> I'll email stub for the dba report
 150 <matsubara> [action] matsubara to email stub asking the dba report
 151 <MootBot> ACTION received:  matsubara to email stub asking the dba report
 152 <matsubara> [topic] QA Stats of the Week
 153 <MootBot> New Topic:  QA Stats of the Week
 154 <matsubara> Soyuz (soyuz): 70
 155 <matsubara> launchpadlib  (launchpadlib): 25
 156 <matsubara> OOPS Tools (oops-tools): 21
 157 <matsubara> Launchpad Developer Utilities (lp-dev-utils): 9
 158 <matsubara> Launchpad Bugs (malone): 8
 159 <matsubara> Launchpad Development Wiki Moin theme (launchpad-dev-moin-theme): 5
 160 <matsubara> Launchpad Auto Build System (launchpad-buildd): 5
 161 <matsubara> Launchpad Buildbot Configuration (lpbuildbot): 2
 162 <matsubara> those are the launchpad projects with untriaged bugs (new state) 
 163 <bigjools> I'd like to point out that most of the soyuz ones are >2 years old
 164 <matsubara> close them all or mark them as triage low :-)
 165 <matsubara> triaged, I mean
 166 <bigjools> I do so when I have time
 167 <matsubara> thanks bigjools
 168 <bigjools> I spend more time triaging new ones
 169 <matsubara> cool
 170 <matsubara> Also please take a look at https://wiki.canonical.com/Launchpad/QATeam/OrphanedCommits/10.03-db-devel and https://wiki.canonical.com/Launchpad/QATeam/OrphanedCommits/10.03-devel
 171 <sinzui> bigjools: if you have ignored the issue for more than 6 months, it is 99% of the time low/triaged. the remaining 1% are security issues we should have fixed
 172 <matsubara> those are the landings for the 10.03 cycle which are not tracked in the http://people.canonical.com/~lpqateam/test-plan-report-10.03.html burndown
 173 <bigjools> sinzui: many of them are simply not bugs any more, in fact
 174 <sinzui> bigjools: yep
 175 * sinzui closed 1/3 of the bugs as fixed that were supposed to go to launchpad-web
 176 <matsubara> [TOPIC] * Proposed items
 177 <MootBot> New Topic:  * Proposed items
 178 <matsubara> there's no new proposed items
 179 <matsubara> anything else before I close?
 180 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda  for the logs. 
 181 <matsubara> #endmeeting