1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <stub> me
   8 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   9 <mbarnett> me
  10 <Chex> me
  11 <henninge> me
  12 <Ursinha> me
  13 <matsubara> rockstar, allenap: hi
  14 <allenap> me
  15 <matsubara> noodles775, hi, can you stand in for bigjools ?
  16 <Ursinha> allenap, are you here in behalf of bugs team?
  17 <noodles775> matsubara: yep.
  18 <matsubara> Ursinha, allenap is the new QA contact
  19 <allenap> Ursinha: Yes.
  20 <Ursinha> cool
  21 <Ursinha> thanks
  22 <matsubara> thanks noodles775 
  23 <matsubara> we're misssing someone from code and registry. let's move on and they can join in later
  24 <matsubara> [TOPIC] Agenda 
  25 <MootBot> New Topic:  Agenda
  26 <matsubara>  * Actions from last meeting
  27 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  28 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  29 <matsubara>  * DBA report (stub)
  30 <matsubara>  * Proposed items
  31 <matsubara> [TOPIC] * Actions from last meeting
  32 <MootBot> New Topic:  * Actions from last meeting
  33 <matsubara>     * barry to continue debug on bug 403606 after finishing 3.0 UI stuff
  34 <matsubara>     * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  35 <matsubara>     * matsubara to update translations qa contact on meeting agenda page
  36 <matsubara>         * done
  37 <matsubara>     * matsubara to generate new oops summary including oopses only for after the rollout
  38 <matsubara>         * done
  39 <ubottu> Launchpad bug 403606 in launchpad-registry "ExpatError errors should be handled to not generate the OOPSes" [High,In progress] https://launchpad.net/bugs/403606
  40 <matsubara> I still haven't trawled those logs. :-( 
  41 <matsubara> [action] * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  42 <MootBot> ACTION received:  * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  43 <matsubara> and I know that barry is working on more debugging for 403606. some code was cowboyed yesterday in the production server to help with this debugging
  44 <matsubara> so I won't re-add that action item
  45 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  46 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  47 <Ursinha> me me
  48 <matsubara> go ahead
  49 <Ursinha> two bugs
  50 <Ursinha> allenap, bug 438985, it's happening quite frequently, do you think it's going to be fixed soon?
  51 <ubottu> Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,Triaged] https://launchpad.net/bugs/438985
  52 <Ursinha> matsubara, who's here for foundations
  53 <Ursinha> ?
  54 <matsubara> I am
  55 <matsubara> what's up?
  56 <Ursinha> :D
  57 <Ursinha> since last meeting we're having a lot of DisconnectionErrors, about 200 a day
  58 <Ursinha> I'd suggest bug 436640 to be fixed and cherrypicked, so we could understand what's happening
  59 <ubottu> Launchpad bug 436640 in launchpad-foundations "Add more info to DisconnectionErrors" [Undecided,New] https://launchpad.net/bugs/436640
  60 <Ursinha> is that hard to accomplish? What do you think?
  61 <allenap> Ursinha: I'll see if we can have a go at it this week.
  62 <Ursinha> matsubara, also, could you triage that
  63 <Ursinha> ?
  64 <Ursinha> pleasse
  65 <Ursinha> please
  66 <Ursinha> thanks allenap
  67 <matsubara> Ursinha, gary was working on it last I heard. I'll ask him once he comes back from the TL meeting
  68 <matsubara> and yes, I'll have it triaged and raise the importance
  69 <Ursinha> matsubara, triaged or in progress?
  70 <matsubara> triaged, I don't know if gary started the patch yet. no point in having a in progress bug without assignee :-)
  71 <matsubara> [action] matsubara to talk to gary about bug 436640
  72 <MootBot> ACTION received:  matsubara to talk to gary about bug 436640
  73 <ubottu> Launchpad bug 436640 in launchpad-foundations "Add more info to DisconnectionErrors" [Undecided,New] https://launchpad.net/bugs/436640
  74 <Ursinha> matsubara, ok, I thought he was actually working on that
  75 <Ursinha> matsubara, thanks
  76 <Ursinha> we have one critical bug, bug 438039, and henninge is already working on it, so ok
  77 <ubottu> Launchpad bug 438039 in rosetta "bzr branch import script oopses sometimes" [Critical,In progress] https://launchpad.net/bugs/438039
  78 <Ursinha> about failing scripts, I'd need rockstar here
  79 <Ursinha> rockstar, what's happening with update_review_diffs?
  80 <matsubara> Ursinha, he showed me some example code to add more info, but I'm not sure if he started it or not. I'll confirm with him before making any changes
  81 <henninge> Ursinha: that has been cherry picked, actually.
  82 * henninge goes to check if the oopses still occur
  83 <Ursinha> henninge, even better :) can you set the bug status properly, please?
  84 <henninge> Ursinha: just done that
  85 <matsubara> Ursinha, please follow up with rockstar about that script after the meeting.
  86 <Ursinha> matsubara, ok, can you add an action item, please?
  87 <Ursinha> thanks henninge :)
  88 <matsubara> [action] Ursinha to talk to rockstar about failing update_preview_diffs script
  89 <MootBot> ACTION received:  Ursinha to talk to rockstar about failing update_preview_diffs script
  90 <Ursinha> thanks :)
  91 <Ursinha> thanks guys
  92 <Ursinha> matsubara, stage is yours again
  93 <matsubara> [action] allenap to talk to Bugs team about bug 438985 and have it fixed.
  94 <MootBot> ACTION received:  allenap to talk to Bugs team about bug 438985 and have it fixed.
  95 <ubottu> Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,Triaged] https://launchpad.net/bugs/438985
  96 <matsubara> thanks Ursinha and everyone
  97 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
  98 <henninge> Ursinha: oopses are still there ... :( Back to 'in progress'
  99 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 100 <Chex> hello everyone, here is the report for this week
 101 <Chex>  Automated LP Cherry Picks: We've migrated to hosting the production LP
 102 <Chex> branches on LP itself and are using PQM to manage production-devel. LP
 103 <Chex> developers can now submit their own Cherry Picks making the deploy of updates
 104 <Chex> easier.
 105 <Chex> - Super-Mirror puller stability: We have setup a nagios check to watch this
 106 <Chex> service, and it has been more stable recently.
 107 <Chex> anyone have any questions??
 108 <matsubara> henninge, when was the fix CP'ed? could it be that those oopses happened before the CP?
 109 * danilos has quit (No route to host)
 110 <noodles775> Chex: yeah, do we just simply pqm-submit targeting production-devel (after getting permission of course)? or is there some documentation?
 111 <noodles775> That sounds great!
 112 <matsubara> indeed
 113 <henninge> matsubara: matsubara unfortunately no, there are some dated today ...
 114 <matsubara> henninge, :-(
 115 <Chex> noodles775: I believe you can just do that pqm-submit , but I would need to check to be sure on that..
 116 <noodles775> Chex: k, thanks!
 117 <matsubara> Chex, could you email the list with your findings and perhaps a link to a wiki page explaining the process?
 118 <Chex> matsubara: yes sure, I will do that.
 119 <matsubara> [action] Chex to email the list about the automated LP cherry pick process
 120 <MootBot> ACTION received:  Chex to email the list about the automated LP cherry pick process
 121 <matsubara> thanks a lot Chex 
 122 <matsubara> moving on
 123 <matsubara> [TOPIC] * DBA report (stub)
 124 <MootBot> New Topic:  * DBA report (stub)
 125 <stub> It looks like we will be moving the master server for the auth replication set to new hardware as part of the continuing migration of the authentication service to the ISD team. Not sure of schedules yet. There will likely need to be a few minutes downtime while we switch masters.
 126 <stub> Load on the databases actually seems a bit lighter with Launchpad 3.0 which is good. No real metrics so this just might be wishful thinking, but I suspect less random statistics scattered around the page and fewer large document reloads due to ajax is helping.
 127 <stub> Nothing else interesting happening.
 128 <matsubara> thanks stub 
 129 <matsubara> [TOPIC] * Proposed items
 130 <MootBot> New Topic:  * Proposed items
 131 <matsubara> no new proposed items
 132 <matsubara> anything else before I close?
 133 <matsubara> 4
 134 <matsubara> 3
 135 <matsubara> 2
 136 <matsubara> 1
 137 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 138 <matsubara> #endmeeting 
 139 <MootBot> Meeting finished at 10:24.