## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 09:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<Ursinha> hai
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<citrus> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
* citrus is now known as sinzui
<sinzui> me
<Ursinha> moi
<gary_poster> me
<matsubara> Chex, rockstar, bigjools, danilos, allenap: hi
<allenap> me
<bigjools> me
<Ursinha> matsubara, rockstar is sprinting, not sure he'll answer
<danilos> me
<matsubara> apologies from stuart. he won't be around today and sent me the dba report
<bigjools> Ursinha: you're talking two more languages now then?
<Ursinha> bigjools, I pretend, and it's working :P
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
<matsubara> * allenap to follow up with pitti about bug 453203
<matsubara> * matsubara to talk to francis about the IncidentLog QA section
<matsubara> * matsubara to email the list about lib/lp/scripts/utilities/sanitizedb.py script
<matsubara>     * email sent
<ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [High,In progress] https://launchpad.net/bugs/453203
<Chex> me
<matsubara> hi Chex 
<Ursinha> matsubara, there's a note in that page about my item
* mbarnett` is now known as mbarnett
<matsubara> I sent the email to the list about the sanitizeddb.py script. I don't think anybody replied 
<matsubara> Ursinha, reloading :-)
<Ursinha> matsubara, I still have to send it, CHR week is kinda consuming :)
<bigjools> matsubara: I did!
<matsubara> [action] * Ursinha to send one email to lp list explaining the qa-tags experiment
<MootBot> ACTION received:  * Ursinha to send one email to lp list explaining the qa-tags experiment
<allenap> matsubara: Re. bug 453203. It's an issue in apport, pitti is going to fix, and he says, from his perspective, that we don't need to work around in Launchpad. However, we may want to anyway.
<ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [High,In progress] https://launchpad.net/bugs/453203
<gary_poster> matsubara: bigjools reviewed the santizedb script and stub has or is going to adjust based on his feedback.  soyuz was stub's biggest worry, so that's cool.
<matsubara> sorry bigjools, I missed your reply. thanks!
<bigjools> :)
<allenap> matsubara: However, any workaround should probably be removed once apport has been fixed.
* salgado (n=salgado@canonical/launchpad/salgado) has joined #launchpad-meeting
<matsubara> I emailed Francis about the IncidentLog QA section and I'll help him out nagging the TL about completing the QA section 
<matsubara> allenap, cool. thanks
<matsubara> I think that's all for this section
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<Ursinha> me
<matsubara> go ahead little bear
<Ursinha> sinzui, bug 481356
<Ursinha> allenap, bug 458180, we're having high timeout counts in lpnet and this one is the main one
<ubottu> Launchpad bug 481356 in launchpad-registry "AttributeError OOPS when trying to subscribe to bugmail of a milestone" [Undecided,New] https://launchpad.net/bugs/481356
<Ursinha> allenap, also, as this is my CHR week, I saw lots of complaints (and some hate mail) about bug 418659
<ubottu> Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180
<Ursinha> is there something we could do?
<ubottu> Launchpad bug 418659 in malone "Reporting duplicate bugs leads to receiving notifications for every duplicate of the original bug" [Undecided,New] https://launchpad.net/bugs/418659
<sinzui> Urshina, that is again a bug link broken by intellectronica's bugmail-ui-fixes
<allenap> Ursinha: Re. the timeout bug, gmb is doing some magic with subscriber lists to help alleviate this.
<Ursinha> allenap, hm, great
<sinzui> I am note certain who should fix it, the diff shows the that this line was removed:
<sinzui> enabled = not IProjectMilestone.providedBy(self.context)
<Ursinha> allenap, can you ask him or comment in the bug, please?
<allenap> Ursinha: I'm sure we can do something about the dupe email one; I'll ask the team.
<allenap> Ursinha: Sure.
<Ursinha> thanks a lot allenap
<Ursinha> that would be much appreciated by lots of users that report ubuntu bugs
<Ursinha> sinzui, well
<Ursinha> sinzui, doesn't seem to hard to fix
<sinzui> It is not hard, every link the milestone menu has to have that line
<Ursinha> sinzui, so, can you fix it? :)
<sinzui> The alternate fix is to remove projectgroups since they break more things then they fix
<sinzui> Ursinha: The bugs team should fix it since it is their bug link.
<Ursinha> sinzui, right then.. allenap?
* allenap reads backwards
* stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
<allenap> sinzui: Do you think the fix is simply to reinstate the line you've mentioned in the bug report?
<sinzui> allenap: no, because intellectronica had icon add/edit handling to it
<allenap> sinzui: Okay, thanks.
<Ursinha> thanks sinzui and allenap
<allenap> Ursinha: I'll pass this to intellectronica. He's at lazr-js this week, UDS next, but perhaps he'll have time anyway :-/
<Ursinha> thanks allenap
<sinzui> My fix would involve removing projectgroups since they break more things then the fix. I would introduce pillar tags
<allenap> sinzui: I'll pass that on.
<Ursinha> we have two critical bugs in progress, one is bug 475394, another is bug 479169
<ubottu> Launchpad bug 475394 in launchpad-code "bzr http access broken after 3.1.10 rollout" [Critical,In progress] https://launchpad.net/bugs/475394
<ubottu> Bug 479169 on http://launchpad.net/bugs/479169 is private
<Ursinha> the bzr bug is code, so we'll have to poke rockstar or abentley to know
<Ursinha> the other is soyuz
<Ursinha> al-maisan is working on it, right bigjools?
<bigjools> Ursinha: it's been rolled out
<Ursinha> bigjools, cherrypicked, you mean?
<bigjools> yes
<Ursinha> bigjools, could you change it to fix released then, please? and add a comment saying that it was cped :)
<bigjools> yes
<Ursinha> matsubara, [action] chase rockstar/abentley about bug 475394
<ubottu> Launchpad bug 475394 in launchpad-code "bzr http access broken after 3.1.10 rollout" [Critical,In progress] https://launchpad.net/bugs/475394
<Ursinha> thanks bigjools :)
* rockstar just logged in
<matsubara> [action] Ursinha to chase rockstar/abentley about bug 475394
<MootBot> ACTION received:  Ursinha to chase rockstar/abentley about bug 475394
<bigjools> anytime :)
<Ursinha> script parse-librarian-apache-logs is failing for some time, is that a real failure?
<rockstar> Ursinha, that's fixed now.
<Ursinha> rockstar, fixed and cherrypicked?
<rockstar> Ursinha, it was cherrypicked to production, I'm not sure if the fix has been landed on trunk.
<rockstar> Ursinha, the code team, with the exception of thumper, is at sprints this week.
<rockstar> Ursinha, looks like the linked branch has been merged.
<rockstar> I marked the bug as "Fix released"
<Ursinha> rockstar, good, could you do the same I've asked bigjools to do, please?
<Ursinha> thanks rockstar :)
<matsubara> [action] cancel previous action item
<MootBot> ACTION received:  cancel previous action item
<Ursinha> fast boy
<Ursinha> about the failing script, who's responsible for parse-librarian-apache-logs?
<matsubara> gary_poster, ^ 
<matsubara> I think salgado is the right person to ask about it. we need to find why it's failing though
<gary_poster> matsubara, Ursinha: I have no idea and have never heard of it, but am happy to acknowledge that it sounds vaguely like foundations :-)  I'll take it.
<sinzui> Was the script failing or just not running
<matsubara> gary_poster, I think salgado wrote that script, IIRC
<Ursinha> my question is similar to sinzui's
<sinzui> The script has definitely been running since the logs were synced
<gary_poster> matsubara: ok.  If you are willing to go straight to him, +1
<salgado> it's failing
<salgado> let me find the logs
<gary_poster> maybe another lovely _pythonpath thing?
<salgado> devpad:/srv/launchpad.net-logs/scripts/loganberry/parse-librarian-apache-access-logs.log
<sinzui> salgado: ok, we did see downloads counts get updated
<salgado> looks like an actual bug in the script
<salgado> sinzui, right, the script parsed all the logs but some recent log lines seem to have broken it
<salgado> oh
<salgado> it might just be trying to parse an empty file
<salgado> just need to ask the losas to get a 'ls -al' of the files in the directory to which we point the script
<salgado> mthaddon, can you do that?
<mthaddon> salgado: what's that?
<matsubara> salgado, in any case we'd need a bug reported to deal with that case right?
<salgado> yeah, definitely looks like an empty file
<salgado> matsubara, right
<Ursinha> salgado, want me to file a bug?
<salgado> mthaddon, 'ls -al <dir-containing-librarian-log-files-to-be-parsed>'
<salgado> Ursinha, please.  I won't have time for filing/fixing it this week
<mthaddon> salgado: https://pastebin.canonical.com/24669/
<matsubara> [action] matsubara to file a bug about the script that parses librarian logs failure
<MootBot> ACTION received:  matsubara to file a bug about the script that parses librarian logs failure
<Ursinha> matsubara, I'm doing it right now
<matsubara> [action] cancel the previous action item
<MootBot> ACTION received:  cancel the previous action item
<matsubara> thanks Ursinha 
<matsubara> Ursinha, anything else for this section?
<Ursinha> matsubara, no, thanks
<salgado> mthaddon, actually need 'ls -al /srv/launchpadlibrarian.net-logs/*/'
<matsubara> thanks everyone.
<Ursinha> salgado, is that unrelated to bug 422552?
<ubottu> Launchpad bug 422552 in launchpad-foundations "Apache log parser failed to parse a line because it had too many values to unpack" [Undecided,Triaged] https://launchpad.net/bugs/422552
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<mthaddon> salgado: https://pastebin.canonical.com/24672/
<salgado> Ursinha, yes, unrelated
<Chex> Hello everyone
<Chex> here is our report for this week:
<Chex> - New LOSA python script to hide Bugs & Questions comments:
<Chex>         ; This is to replace direct DB editing. Chex is working on getting this setup and documented properly.
<Chex> - QA Column of the incident log:
<Chex>         ; This column is still not being updated by anyone; we would appreciate
<Chex>           if people could look at and update this when possible.
<Chex> - LP Incidents of note:
<Chex>         ; Launchpad 3.1.10 major release rolled out on 06-Nov
<Chex>         ; CP 8648 rolled out to loganberry-launchpad, loganberry-bzrsyncd,
<Chex>           codehost, lpnet, edge, cesium, cocoplum, germanium
<Chex>         ; CP 8649 of production-stable rolled out to cesium, germanium,
<Chex>           loganberry-launchpad, loganberry-bzrsyncd, codehost
<matsubara> Chex, I'm help Francis with the nagging about the QA column incident log
<Ursinha> (salgado: bug 481375, could you triage it, please?)
<ubottu> Launchpad bug 481375 in launchpad-foundations "parse-librarian-apache-logs script failing trying to parse empty files" [Undecided,New] https://launchpad.net/bugs/481375
<Chex> matsubara: great, thanks a lot for that
<Chex> and thats our report, any questions for us??
<matsubara> Chex, basically the only problem there is the codebrowse issue which is the same bug as usual (https://bugs.edge.launchpad.net/launchpad-code/+bug/118625)
<ubottu> Launchpad bug 118625 in loggerhead "codebrowse sometimes hangs" [High,Triaged]
<matsubara> Chex, and I have asked gary_poster to investigate the core dump output from the lpnet13 death recently.
<Chex> matsubara: ok, thats great, too.
<matsubara> once more info is known about it, a bug will be reported and added to the column
<stub> I suspect the incident log isn't going to get updated without some prodding as people are generally worrying about today rather than yesterday.
<matsubara> stub, I'll be doing the prodding 
<stub> Thats good - I can't even find the wiki page ;)
<bigjools> it would be good if you made codebrowse changes trivial so I only get emailed about less-comment incidents
<bigjools> (on the incident log wiki I mean)
<Chex> stub: https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
<matsubara> stub, and once we have a bug filed about it, all the discussion can happen in the bug report, soothing losas worries
<matsubara> stub, https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
<bigjools> s/less-comment/less-common/   geez
<matsubara> I think that's all from Chex. thank you
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> A second attempted run of the database sanitization script is underway on production data. We should see how far it gets this time in a few hours (it took over 6 hours to fail last time). This has been increasing load on chokecherry which hosts one of the production slave databases, but I doubt it is noticeable.
<stub> A fix has landed and staging database restores have been reenabled. The current rebuild will fail because the fix hasn't got through to launchpad/stable, so we need to wait until tomorrow to see if things work out as hoped.
<stub> Load has been higher for a while. From eyeball testing, this load is from the appservers. I think this is more to do with increased traffic since we brought the new appservers online rather than code regressions. We may need to start identifying and optimizing heavy db load pages soon. memcache might help here once that branch can land.
<stub> oot
<matsubara> no questions for stub?
<matsubara> all right, thanks stub 
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no new proposed items
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 09:45.