1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 09:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <Ursinha> hai
   6 <matsubara> [TOPIC] Roll Call 
   7 <MootBot> New Topic:  Roll Call
   8 <citrus> me
   9 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
  10 * citrus is now known as sinzui
  11 <sinzui> me
  12 <Ursinha> moi
  13 <gary_poster> me
  14 <matsubara> Chex, rockstar, bigjools, danilos, allenap: hi
  15 <allenap> me
  16 <bigjools> me
  17 <Ursinha> matsubara, rockstar is sprinting, not sure he'll answer
  18 <danilos> me
  19 <matsubara> apologies from stuart. he won't be around today and sent me the dba report
  20 <bigjools> Ursinha: you're talking two more languages now then?
  21 <Ursinha> bigjools, I pretend, and it's working :P
  22 <matsubara> [TOPIC] Agenda 
  23 <MootBot> New Topic:  Agenda
  24 <matsubara>  * Actions from last meeting
  25 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  26 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  27 <matsubara>  * DBA report (stub)
  28 <matsubara>  * Proposed items
  29 <matsubara> [TOPIC] * Actions from last meeting
  30 <MootBot> New Topic:  * Actions from last meeting
  31 <matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
  32 <matsubara> * allenap to follow up with pitti about bug 453203
  33 <matsubara> * matsubara to talk to francis about the IncidentLog QA section
  34 <matsubara> * matsubara to email the list about lib/lp/scripts/utilities/sanitizedb.py script
  35 <matsubara>     * email sent
  36 <ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [High,In progress] https://launchpad.net/bugs/453203
  37 <Chex> me
  38 <matsubara> hi Chex 
  39 <Ursinha> matsubara, there's a note in that page about my item
  40 * mbarnett` is now known as mbarnett
  41 <matsubara> I sent the email to the list about the sanitizeddb.py script. I don't think anybody replied 
  42 <matsubara> Ursinha, reloading :-)
  43 <Ursinha> matsubara, I still have to send it, CHR week is kinda consuming :)
  44 <bigjools> matsubara: I did!
  45 <matsubara> [action] * Ursinha to send one email to lp list explaining the qa-tags experiment
  46 <MootBot> ACTION received:  * Ursinha to send one email to lp list explaining the qa-tags experiment
  47 <allenap> matsubara: Re. bug 453203. It's an issue in apport, pitti is going to fix, and he says, from his perspective, that we don't need to work around in Launchpad. However, we may want to anyway.
  48 <ubottu> Launchpad bug 453203 in apport "UnicodeDecodeError in +filebug: unexpected code byte" [High,In progress] https://launchpad.net/bugs/453203
  49 <gary_poster> matsubara: bigjools reviewed the santizedb script and stub has or is going to adjust based on his feedback.  soyuz was stub's biggest worry, so that's cool.
  50 <matsubara> sorry bigjools, I missed your reply. thanks!
  51 <bigjools> :)
  52 <allenap> matsubara: However, any workaround should probably be removed once apport has been fixed.
  53 * salgado (n=salgado@canonical/launchpad/salgado) has joined #launchpad-meeting
  54 <matsubara> I emailed Francis about the IncidentLog QA section and I'll help him out nagging the TL about completing the QA section 
  55 <matsubara> allenap, cool. thanks
  56 <matsubara> I think that's all for this section
  57 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  58 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  59 <Ursinha> me
  60 <matsubara> go ahead little bear
  61 <Ursinha> sinzui, bug 481356
  62 <Ursinha> allenap, bug 458180, we're having high timeout counts in lpnet and this one is the main one
  63 <ubottu> Launchpad bug 481356 in launchpad-registry "AttributeError OOPS when trying to subscribe to bugmail of a milestone" [Undecided,New] https://launchpad.net/bugs/481356
  64 <Ursinha> allenap, also, as this is my CHR week, I saw lots of complaints (and some hate mail) about bug 418659
  65 <ubottu> Launchpad bug 458180 in malone "BugTask:+index timing out" [High,Triaged] https://launchpad.net/bugs/458180
  66 <Ursinha> is there something we could do?
  67 <ubottu> Launchpad bug 418659 in malone "Reporting duplicate bugs leads to receiving notifications for every duplicate of the original bug" [Undecided,New] https://launchpad.net/bugs/418659
  68 <sinzui> Urshina, that is again a bug link broken by intellectronica's bugmail-ui-fixes
  69 <allenap> Ursinha: Re. the timeout bug, gmb is doing some magic with subscriber lists to help alleviate this.
  70 <Ursinha> allenap, hm, great
  71 <sinzui> I am note certain who should fix it, the diff shows the that this line was removed:
  72 <sinzui> enabled = not IProjectMilestone.providedBy(self.context)
  73 <Ursinha> allenap, can you ask him or comment in the bug, please?
  74 <allenap> Ursinha: I'm sure we can do something about the dupe email one; I'll ask the team.
  75 <allenap> Ursinha: Sure.
  76 <Ursinha> thanks a lot allenap
  77 <Ursinha> that would be much appreciated by lots of users that report ubuntu bugs
  78 <Ursinha> sinzui, well
  79 <Ursinha> sinzui, doesn't seem to hard to fix
  80 <sinzui> It is not hard, every link the milestone menu has to have that line
  81 <Ursinha> sinzui, so, can you fix it? :)
  82 <sinzui> The alternate fix is to remove projectgroups since they break more things then they fix
  83 <sinzui> Ursinha: The bugs team should fix it since it is their bug link.
  84 <Ursinha> sinzui, right then.. allenap?
  85 * allenap reads backwards
  86 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  87 <allenap> sinzui: Do you think the fix is simply to reinstate the line you've mentioned in the bug report?
  88 <sinzui> allenap: no, because intellectronica had icon add/edit handling to it
  89 <allenap> sinzui: Okay, thanks.
  90 <Ursinha> thanks sinzui and allenap
  91 <allenap> Ursinha: I'll pass this to intellectronica. He's at lazr-js this week, UDS next, but perhaps he'll have time anyway :-/
  92 <Ursinha> thanks allenap
  93 <sinzui> My fix would involve removing projectgroups since they break more things then the fix. I would introduce pillar tags
  94 <allenap> sinzui: I'll pass that on.
  95 <Ursinha> we have two critical bugs in progress, one is bug 475394, another is bug 479169
  96 <ubottu> Launchpad bug 475394 in launchpad-code "bzr http access broken after 3.1.10 rollout" [Critical,In progress] https://launchpad.net/bugs/475394
  97 <ubottu> Bug 479169 on http://launchpad.net/bugs/479169 is private
  98 <Ursinha> the bzr bug is code, so we'll have to poke rockstar or abentley to know
  99 <Ursinha> the other is soyuz
 100 <Ursinha> al-maisan is working on it, right bigjools?
 101 <bigjools> Ursinha: it's been rolled out
 102 <Ursinha> bigjools, cherrypicked, you mean?
 103 <bigjools> yes
 104 <Ursinha> bigjools, could you change it to fix released then, please? and add a comment saying that it was cped :)
 105 <bigjools> yes
 106 <Ursinha> matsubara, [action] chase rockstar/abentley about bug 475394
 107 <ubottu> Launchpad bug 475394 in launchpad-code "bzr http access broken after 3.1.10 rollout" [Critical,In progress] https://launchpad.net/bugs/475394
 108 <Ursinha> thanks bigjools :)
 109 * rockstar just logged in
 110 <matsubara> [action] Ursinha to chase rockstar/abentley about bug 475394
 111 <MootBot> ACTION received:  Ursinha to chase rockstar/abentley about bug 475394
 112 <bigjools> anytime :)
 113 <Ursinha> script parse-librarian-apache-logs is failing for some time, is that a real failure?
 114 <rockstar> Ursinha, that's fixed now.
 115 <Ursinha> rockstar, fixed and cherrypicked?
 116 <rockstar> Ursinha, it was cherrypicked to production, I'm not sure if the fix has been landed on trunk.
 117 <rockstar> Ursinha, the code team, with the exception of thumper, is at sprints this week.
 118 <rockstar> Ursinha, looks like the linked branch has been merged.
 119 <rockstar> I marked the bug as "Fix released"
 120 <Ursinha> rockstar, good, could you do the same I've asked bigjools to do, please?
 121 <Ursinha> thanks rockstar :)
 122 <matsubara> [action] cancel previous action item
 123 <MootBot> ACTION received:  cancel previous action item
 124 <Ursinha> fast boy
 125 <Ursinha> about the failing script, who's responsible for parse-librarian-apache-logs?
 126 <matsubara> gary_poster, ^ 
 127 <matsubara> I think salgado is the right person to ask about it. we need to find why it's failing though
 128 <gary_poster> matsubara, Ursinha: I have no idea and have never heard of it, but am happy to acknowledge that it sounds vaguely like foundations :-)  I'll take it.
 129 <sinzui> Was the script failing or just not running
 130 <matsubara> gary_poster, I think salgado wrote that script, IIRC
 131 <Ursinha> my question is similar to sinzui's
 132 <sinzui> The script has definitely been running since the logs were synced
 133 <gary_poster> matsubara: ok.  If you are willing to go straight to him, +1
 134 <salgado> it's failing
 135 <salgado> let me find the logs
 136 <gary_poster> maybe another lovely _pythonpath thing?
 137 <salgado> devpad:/srv/launchpad.net-logs/scripts/loganberry/parse-librarian-apache-access-logs.log
 138 <sinzui> salgado: ok, we did see downloads counts get updated
 139 <salgado> looks like an actual bug in the script
 140 <salgado> sinzui, right, the script parsed all the logs but some recent log lines seem to have broken it
 141 <salgado> oh
 142 <salgado> it might just be trying to parse an empty file
 143 <salgado> just need to ask the losas to get a 'ls -al' of the files in the directory to which we point the script
 144 <salgado> mthaddon, can you do that?
 145 <mthaddon> salgado: what's that?
 146 <matsubara> salgado, in any case we'd need a bug reported to deal with that case right?
 147 <salgado> yeah, definitely looks like an empty file
 148 <salgado> matsubara, right
 149 <Ursinha> salgado, want me to file a bug?
 150 <salgado> mthaddon, 'ls -al <dir-containing-librarian-log-files-to-be-parsed>'
 151 <salgado> Ursinha, please.  I won't have time for filing/fixing it this week
 152 <mthaddon> salgado: https://pastebin.canonical.com/24669/
 153 <matsubara> [action] matsubara to file a bug about the script that parses librarian logs failure
 154 <MootBot> ACTION received:  matsubara to file a bug about the script that parses librarian logs failure
 155 <Ursinha> matsubara, I'm doing it right now
 156 <matsubara> [action] cancel the previous action item
 157 <MootBot> ACTION received:  cancel the previous action item
 158 <matsubara> thanks Ursinha 
 159 <matsubara> Ursinha, anything else for this section?
 160 <Ursinha> matsubara, no, thanks
 161 <salgado> mthaddon, actually need 'ls -al /srv/launchpadlibrarian.net-logs/*/'
 162 <matsubara> thanks everyone.
 163 <Ursinha> salgado, is that unrelated to bug 422552?
 164 <ubottu> Launchpad bug 422552 in launchpad-foundations "Apache log parser failed to parse a line because it had too many values to unpack" [Undecided,Triaged] https://launchpad.net/bugs/422552
 165 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 166 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 167 <mthaddon> salgado: https://pastebin.canonical.com/24672/
 168 <salgado> Ursinha, yes, unrelated
 169 <Chex> Hello everyone
 170 <Chex> here is our report for this week:
 171 <Chex> - New LOSA python script to hide Bugs & Questions comments:
 172 <Chex>         ; This is to replace direct DB editing. Chex is working on getting this setup and documented properly.
 173 <Chex> - QA Column of the incident log:
 174 <Chex>         ; This column is still not being updated by anyone; we would appreciate
 175 <Chex>           if people could look at and update this when possible.
 176 <Chex> - LP Incidents of note:
 177 <Chex>         ; Launchpad 3.1.10 major release rolled out on 06-Nov
 178 <Chex>         ; CP 8648 rolled out to loganberry-launchpad, loganberry-bzrsyncd,
 179 <Chex>           codehost, lpnet, edge, cesium, cocoplum, germanium
 180 <Chex>         ; CP 8649 of production-stable rolled out to cesium, germanium,
 181 <Chex>           loganberry-launchpad, loganberry-bzrsyncd, codehost
 182 <matsubara> Chex, I'm help Francis with the nagging about the QA column incident log
 183 <Ursinha> (salgado: bug 481375, could you triage it, please?)
 184 <ubottu> Launchpad bug 481375 in launchpad-foundations "parse-librarian-apache-logs script failing trying to parse empty files" [Undecided,New] https://launchpad.net/bugs/481375
 185 <Chex> matsubara: great, thanks a lot for that
 186 <Chex> and thats our report, any questions for us??
 187 <matsubara> Chex, basically the only problem there is the codebrowse issue which is the same bug as usual (https://bugs.edge.launchpad.net/launchpad-code/+bug/118625)
 188 <ubottu> Launchpad bug 118625 in loggerhead "codebrowse sometimes hangs" [High,Triaged]
 189 <matsubara> Chex, and I have asked gary_poster to investigate the core dump output from the lpnet13 death recently.
 190 <Chex> matsubara: ok, thats great, too.
 191 <matsubara> once more info is known about it, a bug will be reported and added to the column
 192 <stub> I suspect the incident log isn't going to get updated without some prodding as people are generally worrying about today rather than yesterday.
 193 <matsubara> stub, I'll be doing the prodding 
 194 <stub> Thats good - I can't even find the wiki page ;)
 195 <bigjools> it would be good if you made codebrowse changes trivial so I only get emailed about less-comment incidents
 196 <bigjools> (on the incident log wiki I mean)
 197 <Chex> stub: https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
 198 <matsubara> stub, and once we have a bug filed about it, all the discussion can happen in the bug report, soothing losas worries
 199 <matsubara> stub, https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
 200 <bigjools> s/less-comment/less-common/   geez
 201 <matsubara> I think that's all from Chex. thank you
 202 <matsubara> [TOPIC] * DBA report (stub)
 203 <MootBot> New Topic:  * DBA report (stub)
 204 <stub> A second attempted run of the database sanitization script is underway on production data. We should see how far it gets this time in a few hours (it took over 6 hours to fail last time). This has been increasing load on chokecherry which hosts one of the production slave databases, but I doubt it is noticeable.
 205 <stub> A fix has landed and staging database restores have been reenabled. The current rebuild will fail because the fix hasn't got through to launchpad/stable, so we need to wait until tomorrow to see if things work out as hoped.
 206 <stub> Load has been higher for a while. From eyeball testing, this load is from the appservers. I think this is more to do with increased traffic since we brought the new appservers online rather than code regressions. We may need to start identifying and optimizing heavy db load pages soon. memcache might help here once that branch can land.
 207 <stub> oot
 208 <matsubara> no questions for stub?
 209 <matsubara> all right, thanks stub 
 210 <matsubara> [TOPIC] * Proposed items
 211 <MootBot> New Topic:  * Proposed items
 212 <matsubara> no new proposed items
 213 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 214 <matsubara> #endmeeting 
 215 <MootBot> Meeting finished at 09:45.