1 [12:01] <Ursinha> #startmeeting
   2 [12:01] <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   3 [12:01] <MootBot> Meeting started at 10:01. The chair is Ursinha.
   4 [12:01] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   5 [12:01] <Ursinha> [TOPIC] Roll Call
   6 [12:01] <Ursinha> apologies from matsubara, he's unwell today
   7 [12:01] <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
   8 [12:01] <MootBot> New Topic:  Roll Call
   9 [12:01] <allenap> me
  10 [12:01] <sinzui> me
  11 [12:01] <Ursinha> me!
  12 [12:01] <Chex> salut
  13 [12:02] * Ursinha is happy, karmic koala is out :)
  14 [12:02] --> noodles775_ has joined this channel (n=miken@f053067075.adsl.alicedsl.de).
  15 [12:02] <Ursinha> rockstar, hi
  16 [12:02] <gary_poster> ,e
  17 [12:02] <gary_poster> me
  18 [12:02] <Ursinha> hi noodles775, are you replacing bigjools today?
  19 [12:03] <Ursinha> stub, rockstar are missing
  20 [12:03] <Ursinha> bigjools as well
  21 [12:03] <Ursinha> lets move on, they should join us soon
  22 [12:03] <Ursinha> (or not :P)
  23 [12:03] <Ursinha> [TOPIC] Agenda
  24 [12:03] <Ursinha>  * Actions from last meeting
  25 [12:03] <Ursinha>  * Oops report & Critical Bugs & Broken scripts
  26 [12:03] <Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
  27 [12:03] <Ursinha>  * DBA report (stub)
  28 [12:03] <Ursinha>  * Proposed items
  29 [12:03] <MootBot> New Topic:  Agenda
  30 [12:03] <rockstar> ni
  31 [12:03] <danilos> me
  32 [12:04] <Ursinha> danilos, go rest man, I can do that for us :)
  33 [12:04] <danilos> Ursinha, ok, thanks!
  34 [12:04] <Ursinha> [TOPIC] * Actions from last meeting
  35 [12:04] <Ursinha>  * gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
  36 [12:04] <Ursinha>  * allenap and matsubara to talk about the timeouts on bug pages
  37 [12:04] <Ursinha>  * matsubara to talk to stub about the DBA report when he gets back
  38 [12:04] <MootBot> New Topic:  * Actions from last meeting
  39 [12:04] <ubottu> Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990
  40 [12:04] <Ursinha> allenap, did you? :)
  41 [12:04] <Ursinha> gary_poster, ^
  42 [12:05] <gary_poster> mine: I've been trying to land the pertinent branch for several days, encountering complications and my own mistakes.  I'm trying today too.
  43 [12:05] <Ursinha> thanks gary_poster
  44 [12:05] <allenap> Ursinha: A bit :) There was some discussion about the +index page on the list, but, iirc, it didn't come to any conclusions other than "yes, we'd like to fix it".
  45 [12:06] <Ursinha> [action] gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990
  46 [12:06] <MootBot> ACTION received:  gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990
  47 [12:06] <ubottu> Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990
  48 [12:06] <Ursinha> allenap, ok, right
  49 [12:07] <Ursinha> allenap, BugTask:+index is not the main offender, though it's in the top ten :P
  50 [12:07] <Ursinha> allenap, I'll talk with matsubara to ask him if he took notes somewhere
  51 [12:08] <Ursinha> [action] ask matsubara about his outstanding items from last meeting
  52 [12:08] <MootBot> ACTION received:  ask matsubara about his outstanding items from last meeting
  53 [12:08] --> stub has joined this channel (n=stub@canonical/launchpad/stub).
  54 [12:08] <Ursinha> hi stub :)
  55 [12:08] <allenap> Ursinha: Surely that's the top offender from bugs though?
  56 [12:08] <Ursinha> allenap, I see DistroSeries:+index
  57 [12:09] <allenap> Ursinha: That's Soyuz then :) Not going near that :)
  58 [12:09] <Ursinha> allenap, hehe
  59 [12:09] <Ursinha> it's already being fixed by registry, iirc
  60 [12:09] <sinzui> We landed a fix for DistroSeries:+index but the change is not on edge yet
  61 [12:09] <Ursinha> ha!
  62 [12:09] <Ursinha> :)
  63 [12:09] <Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  64 [12:09] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  65 [12:09] <allenap> Cool. Apologies to Soyuz for tarring them with the Registry brush.
  66 [12:10] <Ursinha> allenap, about bug https://bugs.edge.launchpad.net/malone/+bug/453203, is there something we can do?
  67 [12:10] <Ursinha> sinzui, is https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1396EA55 related to https://bugs.edge.launchpad.net/launchpad-registry/+bug/462742? If not I'll be happy to file a bug
  68 [12:10] <Ursinha> I've already asked henninge to triage https://bugs.edge.launchpad.net/rosetta/+bug/462891
  69 [12:10] <ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [Undecided,New]
  70 [12:10] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1396EA55
  71 [12:10] <ubottu> Launchpad bug 462742 in launchpad-registry "OOPS getting a project index page as an anonymous user" [High,In progress]
  72 [12:10] <ubottu> Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New]
  73 [12:10] <-- noodles775 has left this server (Read error: 104 (Connection reset by peer)).
  74 [12:10] <Ursinha> [action] Ursinha to ask translations to triage bug 462891
  75 [12:10] <MootBot> ACTION received:  Ursinha to ask translations to triage bug 462891
  76 [12:10] <ubottu> Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] https://launchpad.net/bugs/462891
  77 [12:10] <sinzui> Ursinha: that fix was landed by intellectronica yesterday
  78 [12:11] <Ursinha> sinzui, you told me that yesterday, but is that oops related?
  79 [12:11] <sinzui> Ursinha: yes it is the bug subscription link
  80 [12:12] <henninge> sinzui: actually, this is oopsing on trying to access preferredemails/email.
  81 [12:12] <sinzui> Ursinha: sorry. I am wonrg
  82 [12:12] <allenap> Ursinha: I opened a task against apport in Ubuntu. I can follow up with pitti, see if he has ideas. Also, if I have time I'll try to find a way to get the apport data when we next see this error.
  83 [12:12] <Ursinha> oops
  84 [12:12] * Ursinha reverts the change in the bug
  85 [12:13] *** noodles775_ is now known as noodles775.
  86 [12:13] <Ursinha> allenap, that would be much appreciated, thanks 
  87 [12:13] <Ursinha> [action] allenap to follow up with pitti about bug 453203
  88 [12:13] <MootBot> ACTION received:  allenap to follow up with pitti about bug 453203
  89 [12:13] <ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [Undecided,New] https://launchpad.net/bugs/453203
  90 [12:14] <Ursinha> sinzui, want me to file a bug?
  91 [12:15] <sinzui> I will
  92 [12:15] <Ursinha> okay then 
  93 [12:15] <Ursinha> thanks sinzui
  94 [12:15] <Ursinha> and allenap and henninge
  95 [12:16] <Ursinha> henninge, you can triage bug 462891 if you have a moment :)
  96 [12:16] <ubottu> Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] https://launchpad.net/bugs/462891
  97 [12:16] <Ursinha> well
  98 [12:16] <Ursinha> all three critical bugs are already fix committed, good
  99 [12:16] <Ursinha> the jscheck failure is being handled by BjornT, good
 100 [12:16] <henninge> Ursinha: I am on it.
 101 [12:16] <Ursinha> henninge, thank you  :) ice_cream++
 102 [12:17] <Ursinha> moving on
 103 [12:17] <Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 104 [12:17] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 105 [12:17] <Ursinha> hi Chex
 106 [12:17] <Ursinha> :)
 107 [12:17] <Chex> hello
 108 [12:17] <Chex> here is our report for this week
 109 [12:17] <henninge> Ursinha: yeah!
 110 [12:17] <Chex> - LP Ship-it progress in time for Ubuntu release-day:
 111 [12:17] <Chex>         ; New LP app-servers are running
 112 [12:17] <Chex>         ; New LP front-end web servers are running
 113 [12:17] <Chex>         ; New LP Auth DB server has been switched to and is running
 114 [12:17] <Chex>         ; We have a spare LP server available in case of needed extra performance in high loads.
 115 [12:17] <Chex> - LP Incidents of note:
 116 [12:17] <Chex>         ; CP 8536 to lpnet,bzrsyncd
 117 [12:17] <Chex>         ; cowboy config patch to switch auth db master to sweetcherry
 118 [12:17] <Chex>         ; CP 8539 to bzrsyncd, crowberry
 119 [12:18] <Chex> does anyone have any questions??
 120 [12:18] <stub> hackberry going boom might be an incident of note.
 121 [12:18] <Ursinha> Chex, nice to hear we're ready to the ubuntu frenesy :)
 122 [12:19] <Chex> stub: oh ok, neglected to put that on the list
 123 [12:19] <Ursinha> *frenzy
 124 [12:20] * stub frenzies
 125 [12:20] <Ursinha> lol
 126 [12:20] <Ursinha> stub, you can comment on that in your section, that is now :)
 127 [12:20] <Ursinha> any questions for Chex?
 128 [12:20] <sinzui> Ursinha: that is an interesting oops, the create_milestone error only affects an anonymous user on edge.
 129 [12:21] <Ursinha> sinzui, why is it interesting?
 130 [12:21] <sinzui> How often is someone anonymous on edge?
 131 [12:22] <Ursinha> sinzui, sure! 
 132 [12:22] <Ursinha> indeed
 133 [12:22] <Ursinha> following a link, maybe? 
 134 [12:22] <Ursinha> this is indeed weird
 135 [12:23] <Ursinha> well, lets move on just for the sake of the meeting
 136 [12:23] <Ursinha> [TOPIC] * DBA report (stub)
 137 [12:23] <MootBot> New Topic:  * DBA report (stub)
 138 [12:23] <stub> We lost one of the slave databases earlier this week (~05:20 UTC Tuesday). The server load went through the roof. PostgreSQL and shells where still responding despite the load peaking at 94. Processes were not responding to signals.
 139 [12:23] <stub> It turned out one of the drive arrays had hung, so writes were blocking. I'm not sure if this had end user visible results, as the queries made to the slave a read only and likely would have been successfully served from RAM.
 140 [12:23] <stub> The slave was removed from replication as it could not be immediately investigated by IS (not important enough to wake anyone). It was added back into replication the next day.
 141 [12:23] <stub> Staging database updates are currently disabled, as fallout from
 142 [12:23] <stub> moving the authdb replication set master to a new database. A fix for Bug #461800 is up for review which is part of this. I still need to work more on the restoration scripts after this lands.
 143 [12:23] <ubottu> Launchpad bug 461800 in launchpad-foundations "new-slave.py no longer works" [High,In progress] https://launchpad.net/bugs/461800
 144 [12:23] <stub> The new database dedicated to the login service (launchpad_prod_4, lpmain slave and authdb master) is running fine. With this and the new appservers we should hopefully weather release day.
 145 [12:23] <stub> oot.
 146 [12:24] <Ursinha> stub, fixing 461800 will make possible to reenable database updates on staging?
 147 [12:25] <stub> Its part of it, not the whole picture
 148 [12:25] <stub> That allows us to build the slave. I still need to build the original db from a pair of dumps instead of a single dump.
 149 [12:25] <Ursinha> I wonder how and if disabled database updates can affect QA
 150 [12:26] <stub> Generally not adversely.
 151 [12:26] <Ursinha> right
 152 [12:27] <Ursinha> stub, do you have an estimated time to finish that?
 153 [12:28] <stub> I want to have them working again by Monday. It depends if anything explodes from the release I guess.
 154 [12:28] <Ursinha> stub, right, good
 155 [12:28] <Ursinha> thanks
 156 [12:28] <Ursinha> does anyone have something for stub?
 157 [12:28] <Ursinha> all right
 158 [12:29] <Ursinha> Chex, a question, do you know when will PQM close?
 159 [12:29] <Ursinha> perhaps the release manage knows it better, noodles775, hi :)
 160 [12:29] <Ursinha> *manager
 161 [12:30] <noodles775> Ursinha, Chex: according to the docs, 00:00 UTC on Monday.
 162 [12:30] <Ursinha> noodles775, on Monday, right
 163 [12:30] <Ursinha> thanks :)
 164 [12:30] <Chex> Ursinha: ok, good to know then
 165 [12:31] <Ursinha> [TOPIC] * Proposed items
 166 [12:31] <Ursinha> I have something
 167 [12:31] <Ursinha> more of a reminder :)
 168 [12:31] <Ursinha> don't forget about QAing your stuff, it's now or never for next release! :) Needing help, ping the QA-person-in-charge and we'll be glad to help
 169 [12:31] <MootBot> New Topic:  * Proposed items
 170 [12:31] <Ursinha> does anyone else have something?
 171 [12:31] <noodles775> Quite a few teams with > 15 items waiting for QA:
 172 [12:31] <noodles775> http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html
 173 [12:31] <MootBot> LINK received:  http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html
 174 [12:32] *** salgado is now known as salgado-afk.
 175 [12:33] <Ursinha> noodles775, translations doesn't count, we're trying to use lp as QA tool, tagging bugs as qa-needstesting, qa-ok and so on
 176 [12:33] <Ursinha> I'll send one email to the list explaining that today
 177 [12:33] <noodles775> Ursinha: great, thanks.
 178 [12:33] <Ursinha> [action] Ursinha to send one email to lp list explaining the qa-tags experiment
 179 [12:33] <MootBot> ACTION received:  Ursinha to send one email to lp list explaining the qa-tags experiment
 180 [12:33] <Ursinha> thanks for the reminder noodles775 :)
 181 [12:33] <Ursinha> right, anyone else?
 182 [12:34] <Ursinha> good!
 183 [12:34] <gary_poster> :-)
 184 [12:34] <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
 185 [12:34] <Ursinha> #endmeeting 
 186 [12:34] <MootBot> Meeting finished at 10:34.