#format irc
[12:01] <Ursinha> #startmeeting
[12:01] <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
[12:01] <MootBot> Meeting started at 10:01. The chair is Ursinha.
[12:01] <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
[12:01] <Ursinha> [TOPIC] Roll Call
[12:01] <Ursinha> apologies from matsubara, he's unwell today
[12:01] <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
[12:01] <MootBot> New Topic:  Roll Call
[12:01] <allenap> me
[12:01] <sinzui> me
[12:01] <Ursinha> me!
[12:01] <Chex> salut
[12:02] * Ursinha is happy, karmic koala is out :)
[12:02] --> noodles775_ has joined this channel (n=miken@f053067075.adsl.alicedsl.de).
[12:02] <Ursinha> rockstar, hi
[12:02] <gary_poster> ,e
[12:02] <gary_poster> me
[12:02] <Ursinha> hi noodles775, are you replacing bigjools today?
[12:03] <Ursinha> stub, rockstar are missing
[12:03] <Ursinha> bigjools as well
[12:03] <Ursinha> lets move on, they should join us soon
[12:03] <Ursinha> (or not :P)
[12:03] <Ursinha> [TOPIC] Agenda
[12:03] <Ursinha>  * Actions from last meeting
[12:03] <Ursinha>  * Oops report & Critical Bugs & Broken scripts
[12:03] <Ursinha>  * Operations report (mthaddon/Chex/spm/mbarnett)
[12:03] <Ursinha>  * DBA report (stub)
[12:03] <Ursinha>  * Proposed items
[12:03] <MootBot> New Topic:  Agenda
[12:03] <rockstar> ni
[12:03] <danilos> me
[12:04] <Ursinha> danilos, go rest man, I can do that for us :)
[12:04] <danilos> Ursinha, ok, thanks!
[12:04] <Ursinha> [TOPIC] * Actions from last meeting
[12:04] <Ursinha>  * gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
[12:04] <Ursinha>  * allenap and matsubara to talk about the timeouts on bug pages
[12:04] <Ursinha>  * matsubara to talk to stub about the DBA report when he gets back
[12:04] <MootBot> New Topic:  * Actions from last meeting
[12:04] <ubottu> Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990
[12:04] <Ursinha> allenap, did you? :)
[12:04] <Ursinha> gary_poster, ^
[12:05] <gary_poster> mine: I've been trying to land the pertinent branch for several days, encountering complications and my own mistakes.  I'm trying today too.
[12:05] <Ursinha> thanks gary_poster
[12:05] <allenap> Ursinha: A bit :) There was some discussion about the +index page on the list, but, iirc, it didn't come to any conclusions other than "yes, we'd like to fix it".
[12:06] <Ursinha> [action] gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990
[12:06] <MootBot> ACTION received:  gary_poster to try to land branch that cherry picks lazr.restful updates on lpnet for bug 331990
[12:06] <ubottu> Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990
[12:06] <Ursinha> allenap, ok, right
[12:07] <Ursinha> allenap, BugTask:+index is not the main offender, though it's in the top ten :P
[12:07] <Ursinha> allenap, I'll talk with matsubara to ask him if he took notes somewhere
[12:08] <Ursinha> [action] ask matsubara about his outstanding items from last meeting
[12:08] <MootBot> ACTION received:  ask matsubara about his outstanding items from last meeting
[12:08] --> stub has joined this channel (n=stub@canonical/launchpad/stub).
[12:08] <Ursinha> hi stub :)
[12:08] <allenap> Ursinha: Surely that's the top offender from bugs though?
[12:08] <Ursinha> allenap, I see DistroSeries:+index
[12:09] <allenap> Ursinha: That's Soyuz then :) Not going near that :)
[12:09] <Ursinha> allenap, hehe
[12:09] <Ursinha> it's already being fixed by registry, iirc
[12:09] <sinzui> We landed a fix for DistroSeries:+index but the change is not on edge yet
[12:09] <Ursinha> ha!
[12:09] <Ursinha> :)
[12:09] <Ursinha> [TOPIC] * Oops report & Critical Bugs & Broken scripts
[12:09] <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
[12:09] <allenap> Cool. Apologies to Soyuz for tarring them with the Registry brush.
[12:10] <Ursinha> allenap, about bug https://bugs.edge.launchpad.net/malone/+bug/453203, is there something we can do?
[12:10] <Ursinha> sinzui, is https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1396EA55 related to https://bugs.edge.launchpad.net/launchpad-registry/+bug/462742? If not I'll be happy to file a bug
[12:10] <Ursinha> I've already asked henninge to triage https://bugs.edge.launchpad.net/rosetta/+bug/462891
[12:10] <ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [Undecided,New]
[12:10] <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1396EA55
[12:10] <ubottu> Launchpad bug 462742 in launchpad-registry "OOPS getting a project index page as an anonymous user" [High,In progress]
[12:10] <ubottu> Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New]
[12:10] <-- noodles775 has left this server (Read error: 104 (Connection reset by peer)).
[12:10] <Ursinha> [action] Ursinha to ask translations to triage bug 462891
[12:10] <MootBot> ACTION received:  Ursinha to ask translations to triage bug 462891
[12:10] <ubottu> Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] https://launchpad.net/bugs/462891
[12:10] <sinzui> Ursinha: that fix was landed by intellectronica yesterday
[12:11] <Ursinha> sinzui, you told me that yesterday, but is that oops related?
[12:11] <sinzui> Ursinha: yes it is the bug subscription link
[12:12] <henninge> sinzui: actually, this is oopsing on trying to access preferredemails/email.
[12:12] <sinzui> Ursinha: sorry. I am wonrg
[12:12] <allenap> Ursinha: I opened a task against apport in Ubuntu. I can follow up with pitti, see if he has ideas. Also, if I have time I'll try to find a way to get the apport data when we next see this error.
[12:12] <Ursinha> oops
[12:12] * Ursinha reverts the change in the bug
[12:13] *** noodles775_ is now known as noodles775.
[12:13] <Ursinha> allenap, that would be much appreciated, thanks 
[12:13] <Ursinha> [action] allenap to follow up with pitti about bug 453203
[12:13] <MootBot> ACTION received:  allenap to follow up with pitti about bug 453203
[12:13] <ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [Undecided,New] https://launchpad.net/bugs/453203
[12:14] <Ursinha> sinzui, want me to file a bug?
[12:15] <sinzui> I will
[12:15] <Ursinha> okay then 
[12:15] <Ursinha> thanks sinzui
[12:15] <Ursinha> and allenap and henninge
[12:16] <Ursinha> henninge, you can triage bug 462891 if you have a moment :)
[12:16] <ubottu> Launchpad bug 462891 in rosetta "TraversalError on +export" [Undecided,New] https://launchpad.net/bugs/462891
[12:16] <Ursinha> well
[12:16] <Ursinha> all three critical bugs are already fix committed, good
[12:16] <Ursinha> the jscheck failure is being handled by BjornT, good
[12:16] <henninge> Ursinha: I am on it.
[12:16] <Ursinha> henninge, thank you  :) ice_cream++
[12:17] <Ursinha> moving on
[12:17] <Ursinha> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
[12:17] <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
[12:17] <Ursinha> hi Chex
[12:17] <Ursinha> :)
[12:17] <Chex> hello
[12:17] <Chex> here is our report for this week
[12:17] <henninge> Ursinha: yeah!
[12:17] <Chex> - LP Ship-it progress in time for Ubuntu release-day:
[12:17] <Chex>         ; New LP app-servers are running
[12:17] <Chex>         ; New LP front-end web servers are running
[12:17] <Chex>         ; New LP Auth DB server has been switched to and is running
[12:17] <Chex>         ; We have a spare LP server available in case of needed extra performance in high loads.
[12:17] <Chex> - LP Incidents of note:
[12:17] <Chex>         ; CP 8536 to lpnet,bzrsyncd
[12:17] <Chex>         ; cowboy config patch to switch auth db master to sweetcherry
[12:17] <Chex>         ; CP 8539 to bzrsyncd, crowberry
[12:18] <Chex> does anyone have any questions??
[12:18] <stub> hackberry going boom might be an incident of note.
[12:18] <Ursinha> Chex, nice to hear we're ready to the ubuntu frenesy :)
[12:19] <Chex> stub: oh ok, neglected to put that on the list
[12:19] <Ursinha> *frenzy
[12:20] * stub frenzies
[12:20] <Ursinha> lol
[12:20] <Ursinha> stub, you can comment on that in your section, that is now :)
[12:20] <Ursinha> any questions for Chex?
[12:20] <sinzui> Ursinha: that is an interesting oops, the create_milestone error only affects an anonymous user on edge.
[12:21] <Ursinha> sinzui, why is it interesting?
[12:21] <sinzui> How often is someone anonymous on edge?
[12:22] <Ursinha> sinzui, sure! 
[12:22] <Ursinha> indeed
[12:22] <Ursinha> following a link, maybe? 
[12:22] <Ursinha> this is indeed weird
[12:23] <Ursinha> well, lets move on just for the sake of the meeting
[12:23] <Ursinha> [TOPIC] * DBA report (stub)
[12:23] <MootBot> New Topic:  * DBA report (stub)
[12:23] <stub> We lost one of the slave databases earlier this week (~05:20 UTC Tuesday). The server load went through the roof. PostgreSQL and shells where still responding despite the load peaking at 94. Processes were not responding to signals.
[12:23] <stub> It turned out one of the drive arrays had hung, so writes were blocking. I'm not sure if this had end user visible results, as the queries made to the slave a read only and likely would have been successfully served from RAM.
[12:23] <stub> The slave was removed from replication as it could not be immediately investigated by IS (not important enough to wake anyone). It was added back into replication the next day.
[12:23] <stub> Staging database updates are currently disabled, as fallout from
[12:23] <stub> moving the authdb replication set master to a new database. A fix for Bug #461800 is up for review which is part of this. I still need to work more on the restoration scripts after this lands.
[12:23] <ubottu> Launchpad bug 461800 in launchpad-foundations "new-slave.py no longer works" [High,In progress] https://launchpad.net/bugs/461800
[12:23] <stub> The new database dedicated to the login service (launchpad_prod_4, lpmain slave and authdb master) is running fine. With this and the new appservers we should hopefully weather release day.
[12:23] <stub> oot.
[12:24] <Ursinha> stub, fixing 461800 will make possible to reenable database updates on staging?
[12:25] <stub> Its part of it, not the whole picture
[12:25] <stub> That allows us to build the slave. I still need to build the original db from a pair of dumps instead of a single dump.
[12:25] <Ursinha> I wonder how and if disabled database updates can affect QA
[12:26] <stub> Generally not adversely.
[12:26] <Ursinha> right
[12:27] <Ursinha> stub, do you have an estimated time to finish that?
[12:28] <stub> I want to have them working again by Monday. It depends if anything explodes from the release I guess.
[12:28] <Ursinha> stub, right, good
[12:28] <Ursinha> thanks
[12:28] <Ursinha> does anyone have something for stub?
[12:28] <Ursinha> all right
[12:29] <Ursinha> Chex, a question, do you know when will PQM close?
[12:29] <Ursinha> perhaps the release manage knows it better, noodles775, hi :)
[12:29] <Ursinha> *manager
[12:30] <noodles775> Ursinha, Chex: according to the docs, 00:00 UTC on Monday.
[12:30] <Ursinha> noodles775, on Monday, right
[12:30] <Ursinha> thanks :)
[12:30] <Chex> Ursinha: ok, good to know then
[12:31] <Ursinha> [TOPIC] * Proposed items
[12:31] <Ursinha> I have something
[12:31] <Ursinha> more of a reminder :)
[12:31] <Ursinha> don't forget about QAing your stuff, it's now or never for next release! :) Needing help, ping the QA-person-in-charge and we'll be glad to help
[12:31] <MootBot> New Topic:  * Proposed items
[12:31] <Ursinha> does anyone else have something?
[12:31] <noodles775> Quite a few teams with > 15 items waiting for QA:
[12:31] <noodles775> http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html
[12:31] <MootBot> LINK received:  http://people.canonical.com/~lpqateam/test-plan-report-3.1.10.html
[12:32] *** salgado is now known as salgado-afk.
[12:33] <Ursinha> noodles775, translations doesn't count, we're trying to use lp as QA tool, tagging bugs as qa-needstesting, qa-ok and so on
[12:33] <Ursinha> I'll send one email to the list explaining that today
[12:33] <noodles775> Ursinha: great, thanks.
[12:33] <Ursinha> [action] Ursinha to send one email to lp list explaining the qa-tags experiment
[12:33] <MootBot> ACTION received:  Ursinha to send one email to lp list explaining the qa-tags experiment
[12:33] <Ursinha> thanks for the reminder noodles775 :)
[12:33] <Ursinha> right, anyone else?
[12:34] <Ursinha> good!
[12:34] <gary_poster> :-)
[12:34] <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
[12:34] <Ursinha> #endmeeting 
[12:34] <MootBot> Meeting finished at 10:34.