## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<gary-sprint> me
<gary-sprint> :-D
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<sinzui> me
<gary-sprint> uh, me
<Ursinha> me
<matsubara> rockstar, Chex, bigjools, allenap: hi
<allenap> me
<matsubara> apologies from stub
<rockstar> me
<mthaddon> me
<bigjools> me
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<mthaddon> matsubara: I'm sitting in for Chex this meeting as he's working on U1 stuff
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> thanks mthaddon 
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas
<matsubara> * Chex to check app server logs and apache logs to see if it can shed any light in the high load issue.
<matsubara> * adeuring to check with gmb about checkwatches failure
<matsubara> * danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance
<matsubara> * bigjools to investigate update-cache failure and reply back to the list
<ubottu> Launchpad bug 438039 in rosetta "bzr branch import script oopses sometimes" [Critical,Fix released] https://launchpad.net/bugs/438039
<danilos> matsubara: the bug tells you what it was :)
<danilos> oh, I forgot to 'me' myself
<matsubara> I'll finish up my action today 
<matsubara> thanks danilos 
<bigjools> I chatted to curtis and as far as wecan tell it was caused by something else holding a transaction/table open
<bigjools> not much I can do
<matsubara> gmb replied to checkwatches failure email. it was a hung process which was killed and service resumed 
<sinzui> Since the PRF ran the following days, I believe it was a long running process that worried our watching proc
<matsubara> bigjools, thanks for checking. I don't see new emails from that script failing so I take it's working normally
<bigjools> yep
<matsubara> mthaddon, any luck investigating the high loading issue?
<matsubara> s/loading/load/
<mthaddon> matsubara: I wasn't aware that was something we were following up on - not sure what the latest is, but I guess part of it plays into the new SplitIt stuff
<mthaddon> i.e. we've just brought a whole bunch of new servers online so we need to see what effect this has on the overall load of the system
<matsubara> all right. I'll take that item off the list and if high load shows up in the graphs we can pursue further
<mthaddon> k
<matsubara> thanks all, moving on
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> it'sme
<Ursinha> gary_poster, bug 331990, can we CP it?
* sinzui stares at Ursinha
<ubottu> Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990
<Ursinha> s/gary_poster/gary-sprint/
<gary-sprint> Ursinha: I do not have CP-foo.
<Ursinha> allenap, can we have a fix for bug 438802 and maybe CP it?
<matsubara> gary-sprint, is this a matter of updating the lazr.restful lib used by lpnet?
<Ursinha> allenap, also, we have bug 438985, it's in progress but without activity for a some time
<ubottu> Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802
<Ursinha> allenap, and bug 458180, that's BugTask index timeouts
<ubottu> Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985
<ubottu> Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180
<Ursinha> sinzui, I've filed bug 458169 and bug 458189, the timeouts on Milestone and DistroSeries index pages
<Ursinha> rockstar, can we have a fix for bug 442981?
<ubottu> Launchpad bug 458169 in launchpad-registry "Distroseries:+index page timing out" [High,Triaged] https://launchpad.net/bugs/458169
<ubottu> Launchpad bug 458189 in launchpad-registry "Milestone:+index pages timing out" [Undecided,New] https://launchpad.net/bugs/458189
<ubottu> Launchpad bug 442981 in launchpad-code "launchpad-project/+activereviews is OOPSing with TypeError (dup-of: 457541)" [High,Triaged] https://launchpad.net/bugs/442981
<ubottu> Launchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/457541
<gary-sprint> Ursinha: maybe I misunderstood.  are you asking for CP-blessing or for CP-shepherding?  If the latter, sure, we can shepherd.
<Ursinha> gary-sprint, shepherding
<sinzui> Ursinha: I replied that I beleive they are dups of 455812
<sinzui> brad is already working on it
<Ursinha> bug 455812
<ubottu> Launchpad bug 455812 in launchpad-registry "distroseries milestone timeout" [High,Triaged] https://launchpad.net/bugs/455812
<Ursinha> hmmm
<gary-sprint> matsubara: not sure, will ask leonardr.
<Ursinha> sinzui,I'll mark it as a dupe then, thanks
<sinzui> not yet
<rockstar> Ursinha, the fix for that bug is closing it as a duplicate of bug 457541
<ubottu> Launchpad bug 457541 in launchpad-code "Active code reviews for Loggerhead OOPSes on edge" [High,Fix released] https://launchpad.net/bugs/457541
<Ursinha> oh
<matsubara> [action] gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
<rockstar> Ursinha, also, that bug is Fix Released.
<MootBot> ACTION received:  gary to talk to leonardr about cherry picking lazr.restful updates on lpnet for bug 331990
<ubottu> Launchpad bug 331990 in launchpad-foundations "The inline editor widget reports a JSON error when saving non-ASCII characters" [High,Fix committed] https://launchpad.net/bugs/331990
<gary-sprint> +1
<Ursinha> rockstar,it still happens, how come?
<rockstar> Ursinha, so yes, you may have it before it was asked.
<sinzui> I have assign the distroseries +index to edwin. I think EdwinGrubbs and bac will find this is the same problem
<sinzui> The oopses of the two new bugs look the the oopses I have been tracking in the older bug
<rockstar> Ursinha, doesn't oops for me.
<Ursinha> rockstar, so the summaries are lying :)
<rockstar> Ursinha, does this url oops for you? https://edge.launchpad.net/launchpad-project/+activereviews
<Ursinha> rockstar, well, it's loading... I'll keep my eye on it and if needed reopen it, right?
<Ursinha> rockstar, thanks
<Ursinha> allenap, hi :)
<allenap> Ursinha: I talk to deryck about getting bug 438802 fixed, and gmb about bug 438985.
<ubottu> Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802
<ubottu> Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985
<Ursinha> allenap, thanks
<allenap> Ursinha: Bug 458180 is a perennial problem.
<ubottu> Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180
<Ursinha> allenap, I see the main offender is bug #1
<ubottu> https://bugs.launchpad.net/ubuntu/+bug/1 (Timeout)
<Ursinha> yes, *sigh*
<allenap> Ursinha: Yeah, it always is :)
<rockstar> Ursinha, yes, but I can't see how it'd get reopened.  It was bad data, we fixededed the database records.
<danilos> Ursinha: and you just made it worse with a reference now :)
<Ursinha> danilos, yes, just to prove my point :P
<matsubara> allenap, it still happens in other bugs too as per jono's email to launchpad-dev about ubuttu timing out
<Ursinha> allenap, there are some oopses not caused by #1
<Ursinha> thanks allenap
<allenap> matsubara: Okay, as someone said, perhaps it's the +text interface.
<Ursinha> gary-sprint, the "buildbot failure in Launchpad on jscheck", is it severe?
<matsubara> allenap, I briefly trawled the summaries and there are a some sofr timeouts on +text, but soft timeouts shouldn't be affecting ubottu
<allenap> matsubara, Ursinha: We need to do something more drastic to get the bug page quicker I think. Caching, etc, and that's coming alone. We've done a lot of the other things we can think of, but I'll discuss it with the team.
<allenap> matsubara: That's interesting.
<allenap> s/alone/along/
<Ursinha> I see some emails from francis and rockstar talking about it, is there something that can or needs to be done?
<matsubara> allenap, perhaps those timeouts are not being logged as OOPSes? similar to 500 we see eventually from apache
<Ursinha> gary-sprint, ^
<matsubara> to the 500 errors I mean
<rockstar> Ursinha, gary-sprint, it is my belief that windmill sucks.
<gary-sprint> Ursinha: it does not appear to be a problem in the basic buildbot setup at the moment.    There are failures in the tests.  This doesn't seem to be a foundations issue AFAICT.  Björn may very well be able to help when he returns
<allenap> matsubara: Okay, I'm not sure what you mean, but we can talk about it after the meeting.
<Ursinha> right gary-sprint, thanks for the info
<matsubara> allenap, sure thing. I'll find the bug I'm referring to
<allenap> matsubara: Thanks.
<matsubara> [action] allenap and matsubara to talk about the timeouts on bug pages
<MootBot> ACTION received:  allenap and matsubara to talk about the timeouts on bug pages
<Ursinha> right, I'm done here
<gary-sprint> rockstar: that's probably a given.  The more interesting question is whether it sucks worse than the alternatives.  My impression is no, but a champion could fight for an alternate view,
<gary-sprint> .
<Ursinha> allenap, is it possible to ask for a cp for bug 438802 when it's fixed?
<ubottu> Launchpad bug 438802 in malone "UnicodeDecodeError changing 'Assigned to' field when summary contains non-ascii" [High,Triaged] https://launchpad.net/bugs/438802
<rockstar> gary-sprint, sadly, there is no better alternative. Windmill sucks less than anything else out there.
* salgado is now known as salgado-lunch
<gary-sprint> :-)
<allenap> Ursinha: Sure.
<Ursinha> anmar was having problems yesterday with bugs with chinese chars, I think it's worth doing a CP
<Ursinha> thanks allenap
<allenap> Ursinha: np, thank you :)
<Ursinha> :)
<matsubara> ok, two fix committed critical bugs 
<matsubara> rockstar, we had some failures on the update_preview_diffs script
<matsubara> on the 19th
<rockstar> matsubara, yeah, we're currently in the process of fixing the various oopses that script creates.
* thumper has quit (Remote closed the connection)
<matsubara> rockstar, ok. can you give me the bug numbers after the meeting?
<rockstar> matsubara, there are many.
<matsubara> gladly we have an oops tag to filter those :-)
<matsubara> rockstar, I'll ping you after the meeting
<Ursinha> thanks everyone
<matsubara> I think that's all for this section. thanks everyone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<mthaddon> SplitIt is the big one this week - now complete with exception of Auth DB split.
<mthaddon> New App servers brought online after haproxy throttlng of connections, we're watching how things are progressing
<mthaddon> A number of CPs done this week
<mthaddon> Is everyone clear on the new CP process?
<mthaddon> Shipit now managed by ISD, and CPs to be approved by nigelp
<mthaddon> Some app servers dying, loggerhead dying, poppy died once - is there a process for reviewing the Incident Log?
<mthaddon> That's about it
<matsubara> mthaddon, last I heard Francis was the one to champion the Incident Log process.
<mthaddon> matsubara: basically we want to be sure someone's reviewing it to look for operational trends in production
<bigjools> did he mention making trvial wiki edits for codebounce so we don't get email for those?
<matsubara> ideally we won't need that codebounce all the time :-)
<Ursinha> matsubara, +1 :)
<mthaddon> bigjools: if we have to get alerts and go through the whole restart, edit wiki nightmare, you can put up with a few wiki edit notifications :)
* matsubara looks at rockstar 
<bigjools> mthaddon: well, no I don't :)
<danilos> mthaddon: the concern is that we may learn to ignore it unless we can filter stuff out *we* can't do anything about
<rockstar> mthaddon, I'm subscribed and get the pleasure of seeing every time you restart loggerhead.
<matsubara> any news about the codebrowser dying all the time?
<bigjools> what danilos said
<danilos> mthaddon: specifically, translations or soyuz team can't help much with codebrowse restarts
<rockstar> matsubara, we are bringing someone on to look into the codebrowse issue.  That's all I know.  We certainly don't have the bandwidth currently to do it.
<mthaddon> fwiw, I usually do trivial that one - I guess maybe the other losas don't - will mention it
<danilos> bigjools: just as an example, and these are very, very common
<sinzui> I believe there is a plan to but people to work on loggerhead
<danilos> mthaddon: in general, anything else shoudn't be a trivial edit, and codebrowse should, that would help old men like bigjools deal with their email :)
<bigjools> ha
<danilos> sinzui: yeah, as flacoste mentioned today, I think we are having a contract that starts today or tomorrow
<matsubara> that's great news
<danilos> mthaddon: but, do note that most team leads are subscribed to LPIncidentLog, and if one isn't, feel free to poke them about it
<mthaddon> danilos: k, thx
<matsubara> that's all mthaddon ?
<mthaddon> yep
<matsubara> all right. thanks everyone
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> stub is on vacation and looks like the db is fine
<matsubara> AFAICT
<matsubara> so let's move on.
<matsubara> [action] matsubara to talk to stub about the DBA report when he gets back
<MootBot> ACTION received:  matsubara to talk to stub about the DBA report when he gets back
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no new proposed items
<matsubara> and I think that's all for today
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting