1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <sinzui> me
   8 <matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
   9 <stub> me
  10 <matsubara> Chex, gary_poster, rockstar, bigjools, hi
  11 * henninge-sprint (n=henning@canonical/launchpad/henninge) has joined #launchpad-meeting
  12 <matsubara> allenap, hi
  13 <gary_poster> me
  14 <henninge-sprint> ignore me
  15 <matsubara> translations are excused today because they're sprinting
  16 <bigjools> me
  17 <allenap> me
  18 <Chex> me
  19 <matsubara> ok, rockstar can join in later
  20 <matsubara> [TOPIC] Agenda 
  21 <MootBot> New Topic:  Agenda
  22 <matsubara>  * Actions from last meeting
  23 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  24 <matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
  25 <matsubara>  * DBA report (stub)
  26 <matsubara>  * Proposed items
  27 <matsubara> [TOPIC] * Actions from last meeting
  28 <MootBot> New Topic:  * Actions from last meeting
  29 <matsubara> * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  30 <matsubara> * matsubara to talk to gary about bug 436640
  31 <matsubara> * Ursinha to talk to rockstar about failing update_preview_diffs script
  32 <matsubara> * allenap to talk to Bugs team about bug 438985 and have it fixed.
  33 <matsubara> * Chex to email the list about the automated LP cherry pick process
  34 <ubottu> Launchpad bug 436640 in oops-tools "Add more info to DisconnectionErrors" [High,In progress] https://launchpad.net/bugs/436640
  35 <ubottu> Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985
  36 <allenap> gmb is working on bug 438985 as of yesterday.
  37 <allenap> He's not here today, so I don't know how that's going.
  38 <matsubara>  I did talk to gary_poster about bug 436640 and I landed a fix for it yesterday.
  39 <matsubara> but I still suck for not trawling the logs...
  40 <gary_poster> yes, yay!
  41 <matsubara> [action] * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  42 <MootBot> ACTION received:  * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
  43 <Ursinha> I did talk to rockstar and abentley and they're already discussing that in another thread
  44 <matsubara> thanks allenap and Ursinha 
  45 <matsubara> I think Francis emailed the list about the new CP process. Is that correct Chex?
  46 <Chex> matsubara: yes I believe so, but the new process is on hold at the moment
  47 <matsubara> Chex, right. that's fine. thanks
  48 <matsubara> let's move on
  49 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  50 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  51 * henninge-sprint has quit ("I outta here!")
  52 <matsubara> OOPS-1374EA1128 for soyuz or foundations
  53 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1374EA1128
  54 * bigjools looks
  55 * gary_poster does too
  56 <matsubara> it's an api operation failing in some librarian thing
  57 <bigjools> that's related to a soyuz bug
  58 <bigjools> bug 443075
  59 <ubottu> Launchpad bug 443075 in soyuz "security package unembargo is broken on edge" [High,In progress] https://launchpad.net/bugs/443075
  60 <bigjools> and is in progress
  61 <matsubara> bigjools, related in the sense that it'll be fixed when a fix for 443075 lands?
  62 <bigjools> yes
  63 <matsubara> cool. thanks
  64 <bigjools> np
  65 <matsubara> and now properly linked to the bug report :-)
  66 <matsubara> allenap, can you confirm OOPS-1371EB1331 is the same as bug 394097
  67 <matsubara> ?
  68 * allenap looks
  69 <ubottu> Launchpad bug 394097 in malone "accessing message 0 of bug 371281 gives wrong return type of IMessage" [Undecided,New] https://launchpad.net/bugs/394097
  70 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1371EB1331
  71 <bigjools> matsubara: what is "properly linked?"
  72 <rockstar> Sorry I'm late...
  73 <matsubara> sinzui, do you know if barry succeeded in CP the expaterror fix?
  74 <sinzui> he informed me that it was up to you to champion that since we know the problem does not affect users
  75 <allenap> matsubara: Yes, they look the same.
  76 <sinzui> matsubara: I believe it is on the CP request list
  77 <matsubara> allenap, abel explains in bug 438671 it's a won't fix. so it means the OOPS is informational only?
  78 <ubottu> Launchpad bug 438671 in checkbox "HWSubmissionMissingFields OOPS on +hwdb/+submit" [Undecided,Confirmed] https://launchpad.net/bugs/438671
  79 <matsubara> thanks allenap 
  80 <matsubara> bigjools, linked the oops to the bug report. oops summaries will contain a link to the bug report from now on 
  81 <bigjools> ah
  82 * ursula (n=ursula@87.250.52.154) has joined #launchpad-meeting
  83 <bigjools> can anyone do that?
  84 * Ursinha has quit (Nick collision from services.)
  85 <matsubara> yep, anyone with access to lp-oops.canonical.com
  86 * ursula is now known as Ursinha
  87 <allenap> matsubara: ISTR that Abel mentioned that on a stand-up; yes these are informational only.
  88 * danilos has quit (Read error: 60 (Operation timed out))
  89 <gary_poster> matsubara: time to introduce "handling" ;-)
  90 <matsubara> sinzui, if it's there, than it's all fine. thanks
  91 <matsubara> gary_poster, yeah, I'm going to announce that :-)
  92 <gary_poster> cool
  93 <matsubara> so, I landed a new api to our ErrorReportingUtility yesterday which marks oops reports as informational only 
  94 <allenap> matsubara: Those OOPSes are more likely to indicate a bug in the client.
  95 <matsubara> I'll email the devel list and let you know more details
  96 <matsubara> [action] matsubara to email the devel list about the new api method
  97 <MootBot> ACTION received:  matsubara to email the devel list about the new api method
  98 <allenap> mars: It might be interesting to use the User-Agent in those OOPSes to file bugs against the clients, i.e. checkbox-gtk.
  99 <matsubara> allenap, right. I'll file a bug to have those marked as informational only
 100 <matsubara> [action] matsubara to file a bug to have the HWSubmissionMissingFields oopses as informational only (note to self: see bug 438671 for more details)
 101 <MootBot> ACTION received:  matsubara to file a bug to have the HWSubmissionMissingFields oopses as informational only (note to self: see bug 438671 for more details)
 102 <ubottu> Launchpad bug 438671 in checkbox "HWSubmissionMissingFields OOPS on +hwdb/+submit" [Undecided,Confirmed] https://launchpad.net/bugs/438671
 103 <matsubara> scripts failures for this week:
 104 <matsubara> the librarian-gc failed 
 105 <matsubara> on the 3rd and the 6th
 106 <matsubara> stub, Chex: do you know anything about it? can you follow up on the failure email please?
 107 <stub> This has now been fixed
 108 <stub> The last run worked fine - required cherry picks have all been made.
 109 <matsubara> cool. thanks stub 
 110 <matsubara> for the critical bugs, we have only one in progress, which is a translations bug.
 111 <matsubara> all the others are fix committed
 112 <matsubara> I think that's all for this section. thanks everyone
 113 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
 114 <MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
 115 <matsubara> Chex, ?
 116 <Chex> hello everyone
 117 <Chex> Here is our LOSA report for this week:
 118 <Chex> - Automated LP Cherry Picks: The automated CP processing into PQM has been disabled for now due to Bug: 442934. We would appreciate to know the status of getting this issue addressed.
 119 <Chex> - SplitIt setup: We have received the new hardware in the Datecentre, and we are in the process of
 120 <Chex> configuring & setting up LP on these new systems now.
 121 <Chex> - New "QA Info" column in the Incident Log: We have a new column in the incident log that the Dev's can
 122 <Chex> fill in to document ways of fixing the production incidents we list.  We suggest that someone reviews the new
 123 <Chex> items on the page and make comments on them at least once a week.
 124 <Chex> - LP incidents of note: Codebrowse restarted many times as usual; CPs applied: CP 9638, 9646 to loganberry ;
 125 <Chex>         CP 9588, 9610, 9621 to librarian2, xmlrpc-internal, bzrsyncd, mailman
 126 * ursula (n=ursula@87.250.52.154) has joined #launchpad-meeting
 127 * Ursinha has quit (Nick collision from services.)
 128 * ursula is now known as Ursinha
 129 <Chex> Does anyone have any questions or comments?
 130 <gary_poster> Chex: flacoste is pretty interested in https://bugs.edge.launchpad.net/pqm/+bug/442934 too .  He has it in his to-do list to pursue.
 131 <ubottu> Launchpad bug 442934 in pqm "unknown url type 'bzr+ssh://bazaar.launchpad.net/$user/$project/$branch;revno=$int'" [Undecided,New]
 132 <matsubara> Chex, about the splitit setup, are we expecting any new oops ids? I glanced over the email about it and notice that there will be lots of new instances running
 133 <matsubara> s/oops ids/oops prefixes/
 134 <gary_poster> (I'm interested too, but he's going to do something about it :-) )
 135 <allenap> Chex: Was that for lp-source-dependencies?
 136 <stub> If I recall from the email conversation, that bug is on a feature we don't actually need to be using.
 137 <stub> (because we have no need to freeze the revision of the dependencies branch being used).
 138 <allenap> Chex: Because... what stub said.
 139 <matsubara> rockstar, any news about the code browse issue?
 140 <Chex> matsubara: not sure about the OOPS ids, but I believe we will need that. I will check on that for you
 141 <rockstar> matsubara, not yet.
 142 <matsubara> [action] chex to check the new oops prefixes added by the splitit setup and inform matsubara
 143 <MootBot> ACTION received:  chex to check the new oops prefixes added by the splitit setup and inform matsubara
 144 <mthaddon> matsubara: for new OOPS ids see the lp-production-configs trunk branch
 145 <Chex> allenap: ok, I will check into that bug status, then. I am not sure of that issue myself.
 146 <gary_poster> stub, allenap, Chex: we need bug 442934 for the branches in sourcecode
 147 <ubottu> Launchpad bug 442934 in pqm "unknown url type 'bzr+ssh://bazaar.launchpad.net/$user/$project/$branch;revno=$int'" [Undecided,New] https://launchpad.net/bugs/442934
 148 <matsubara> thanks mthaddon 
 149 <mthaddon> matsubara: can we 'unaction' that one from Chex?
 150 <gary_poster> not for the buildout-related branch
 151 <matsubara> [action] matsubara to ignore the previous Chex action and look in lp-production-configs for the new oops prefixes.
 152 <MootBot> ACTION received:  matsubara to ignore the previous Chex action and look in lp-production-configs for the new oops prefixes.
 153 <mthaddon> cool, thx
 154 <matsubara> Chex, about the QA info column, do you have a handy link?
 155 <Chex> matsubara: sure, I should have included that
 156 <Chex> https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
 157 <matsubara> thanks Chex 
 158 <matsubara> [action] all QA contacts to inform their teams about the new QA column and what they should do about it.
 159 <MootBot> ACTION received:  all QA contacts to inform their teams about the new QA column and what they should do about it.
 160 <matsubara> Chex, did you send an email about the new QA column to the list?
 161 <matsubara> if not, it'd be great if you could send it and explain what you the losas expects there
 162 <Chex> matsubara: no I did not, but I can do that for you.
 163 <matsubara> thanks Chex 
 164 <Chex> matsubara: sure, no problem
 165 <matsubara> [action] Chex to email the list about the new QA column in https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
 166 <MootBot> ACTION received:  Chex to email the list about the new QA column in https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
 167 <matsubara> let's move on
 168 <matsubara> [TOPIC] * DBA report (stub)
 169 <MootBot> New Topic:  * DBA report (stub)
 170 <stub> The database patch approval process has changed. Previously, they went through me and then had to get a second level of approval by Mark at a monthly meeting. Now they just need to be approved by both myself and jml. We should now be able to track everything with merge proposals and the system will be much less time sensitive, allowing reviews to happen throughout the cycle.
 171 <stub> A new database replica is being brought online as I type. It will be left running as a replica for a few days to ensure it has no problems keeping up (it doesn't have much RAM). Assuming the burn in goes fine, it will become the master for the authdb replication set as part of the ongoing process to separate the authentication systems from Launchpad.
 172 <stub> The databases will all need to be bounced soon to increase the maximum number of connections. We need to do this as we are bringing new appservers on line (per the results of the performance sprint in Montreal). Assuming tests on staging go well, we should be able to do this with no end-user noticeable effects as the database reconnection handling with handle the short outage just fine.
 173 <stub> Bug #426823 is tracking the progress of getting Launchpad to run with PostgreSQL 8.4. At the moment, it looks like we will need to wait until 8.4.2 as we are tickling a PostgreSQL bug. This bug has been tracked down with upstream and a patch is available, but I'd prefer if we stuck to official releases than assembling our own package. We are also blocked by the release of Slony-I 1.2.17, assuming we don't want to move forward with using th
 174 <ubottu> Launchpad bug 426823 in launchpad-foundations "Update to PostgreSQL 8.4" [Medium,In progress] https://launchpad.net/bugs/426823
 175 <stub> There are still some test failures to address with PG 8.4. Some are ordering problems where we were incorrectly relying on database ordering, other issues might be more serious where we are obtaining different results than under 8.3. I may pass some of these onto the teams to see if something is broken under 8.4 or if the issue is benign.
 176 <stub> oot.
 177 <matsubara> questions for stub?
 178 <matsubara> thanks stub 
 179 <matsubara> [TOPIC] * Proposed items
 180 <MootBot> New Topic:  * Proposed items
 181 <matsubara> no new proposed item
 182 <matsubara> anything else before I close?
 183 <matsubara> 4
 184 <matsubara> 3
 185 <matsubara> 2
 186 <matsubara> 1
 187 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
 188 <matsubara> #endmeeting 
 189 <MootBot> Meeting finished at 10:36.
 190 <gary_poster> thanks matsubara!
 191 <allenap> stub: I'm just catching up. One line got truncated, ends with "assuming we don't want to move forward with using th". Can you paste from there?
 192 <stub> assuming we don't want to move forward with using the release candidate. I'll land the branch allowing people to run 'make schema' with PG 8.4, but I don't recommend anyone do this yet because the PG bug we are tickling is fairly pervasive and it will end in frustration.