## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<sinzui> me
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<stub> me
<matsubara> Chex, gary_poster, rockstar, bigjools, hi
* henninge-sprint (n=henning@canonical/launchpad/henninge) has joined #launchpad-meeting
<matsubara> allenap, hi
<gary_poster> me
<henninge-sprint> ignore me
<matsubara> translations are excused today because they're sprinting
<bigjools> me
<allenap> me
<Chex> me
<matsubara> ok, rockstar can join in later
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<matsubara> * matsubara to talk to gary about bug 436640
<matsubara> * Ursinha to talk to rockstar about failing update_preview_diffs script
<matsubara> * allenap to talk to Bugs team about bug 438985 and have it fixed.
<matsubara> * Chex to email the list about the automated LP cherry pick process
<ubottu> Launchpad bug 436640 in oops-tools "Add more info to DisconnectionErrors" [High,In progress] https://launchpad.net/bugs/436640
<ubottu> Launchpad bug 438985 in malone "Trying to make myself as bug supervisor of my project oopses" [High,In progress] https://launchpad.net/bugs/438985
<allenap> gmb is working on bug 438985 as of yesterday.
<allenap> He's not here today, so I don't know how that's going.
<matsubara>  I did talk to gary_poster about bug 436640 and I landed a fix for it yesterday.
<matsubara> but I still suck for not trawling the logs...
<gary_poster> yes, yay!
<matsubara> [action] * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<MootBot> ACTION received:  * matsubara to trawl logs related to high load on edge yesterday and ping Chex about it
<Ursinha> I did talk to rockstar and abentley and they're already discussing that in another thread
<matsubara> thanks allenap and Ursinha 
<matsubara> I think Francis emailed the list about the new CP process. Is that correct Chex?
<Chex> matsubara: yes I believe so, but the new process is on hold at the moment
<matsubara> Chex, right. that's fine. thanks
<matsubara> let's move on
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
* henninge-sprint has quit ("I outta here!")
<matsubara> OOPS-1374EA1128 for soyuz or foundations
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1374EA1128
* bigjools looks
* gary_poster does too
<matsubara> it's an api operation failing in some librarian thing
<bigjools> that's related to a soyuz bug
<bigjools> bug 443075
<ubottu> Launchpad bug 443075 in soyuz "security package unembargo is broken on edge" [High,In progress] https://launchpad.net/bugs/443075
<bigjools> and is in progress
<matsubara> bigjools, related in the sense that it'll be fixed when a fix for 443075 lands?
<bigjools> yes
<matsubara> cool. thanks
<bigjools> np
<matsubara> and now properly linked to the bug report :-)
<matsubara> allenap, can you confirm OOPS-1371EB1331 is the same as bug 394097
<matsubara> ?
* allenap looks
<ubottu> Launchpad bug 394097 in malone "accessing message 0 of bug 371281 gives wrong return type of IMessage" [Undecided,New] https://launchpad.net/bugs/394097
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1371EB1331
<bigjools> matsubara: what is "properly linked?"
<rockstar> Sorry I'm late...
<matsubara> sinzui, do you know if barry succeeded in CP the expaterror fix?
<sinzui> he informed me that it was up to you to champion that since we know the problem does not affect users
<allenap> matsubara: Yes, they look the same.
<sinzui> matsubara: I believe it is on the CP request list
<matsubara> allenap, abel explains in bug 438671 it's a won't fix. so it means the OOPS is informational only?
<ubottu> Launchpad bug 438671 in checkbox "HWSubmissionMissingFields OOPS on +hwdb/+submit" [Undecided,Confirmed] https://launchpad.net/bugs/438671
<matsubara> thanks allenap 
<matsubara> bigjools, linked the oops to the bug report. oops summaries will contain a link to the bug report from now on 
<bigjools> ah
* ursula (n=ursula@87.250.52.154) has joined #launchpad-meeting
<bigjools> can anyone do that?
* Ursinha has quit (Nick collision from services.)
<matsubara> yep, anyone with access to lp-oops.canonical.com
* ursula is now known as Ursinha
<allenap> matsubara: ISTR that Abel mentioned that on a stand-up; yes these are informational only.
* danilos has quit (Read error: 60 (Operation timed out))
<gary_poster> matsubara: time to introduce "handling" ;-)
<matsubara> sinzui, if it's there, than it's all fine. thanks
<matsubara> gary_poster, yeah, I'm going to announce that :-)
<gary_poster> cool
<matsubara> so, I landed a new api to our ErrorReportingUtility yesterday which marks oops reports as informational only 
<allenap> matsubara: Those OOPSes are more likely to indicate a bug in the client.
<matsubara> I'll email the devel list and let you know more details
<matsubara> [action] matsubara to email the devel list about the new api method
<MootBot> ACTION received:  matsubara to email the devel list about the new api method
<allenap> mars: It might be interesting to use the User-Agent in those OOPSes to file bugs against the clients, i.e. checkbox-gtk.
<matsubara> allenap, right. I'll file a bug to have those marked as informational only
<matsubara> [action] matsubara to file a bug to have the HWSubmissionMissingFields oopses as informational only (note to self: see bug 438671 for more details)
<MootBot> ACTION received:  matsubara to file a bug to have the HWSubmissionMissingFields oopses as informational only (note to self: see bug 438671 for more details)
<ubottu> Launchpad bug 438671 in checkbox "HWSubmissionMissingFields OOPS on +hwdb/+submit" [Undecided,Confirmed] https://launchpad.net/bugs/438671
<matsubara> scripts failures for this week:
<matsubara> the librarian-gc failed 
<matsubara> on the 3rd and the 6th
<matsubara> stub, Chex: do you know anything about it? can you follow up on the failure email please?
<stub> This has now been fixed
<stub> The last run worked fine - required cherry picks have all been made.
<matsubara> cool. thanks stub 
<matsubara> for the critical bugs, we have only one in progress, which is a translations bug.
<matsubara> all the others are fix committed
<matsubara> I think that's all for this section. thanks everyone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara> Chex, ?
<Chex> hello everyone
<Chex> Here is our LOSA report for this week:
<Chex> - Automated LP Cherry Picks: The automated CP processing into PQM has been disabled for now due to Bug: 442934. We would appreciate to know the status of getting this issue addressed.
<Chex> - SplitIt setup: We have received the new hardware in the Datecentre, and we are in the process of
<Chex> configuring & setting up LP on these new systems now.
<Chex> - New "QA Info" column in the Incident Log: We have a new column in the incident log that the Dev's can
<Chex> fill in to document ways of fixing the production incidents we list.  We suggest that someone reviews the new
<Chex> items on the page and make comments on them at least once a week.
<Chex> - LP incidents of note: Codebrowse restarted many times as usual; CPs applied: CP 9638, 9646 to loganberry ;
<Chex>         CP 9588, 9610, 9621 to librarian2, xmlrpc-internal, bzrsyncd, mailman
* ursula (n=ursula@87.250.52.154) has joined #launchpad-meeting
* Ursinha has quit (Nick collision from services.)
* ursula is now known as Ursinha
<Chex> Does anyone have any questions or comments?
<gary_poster> Chex: flacoste is pretty interested in https://bugs.edge.launchpad.net/pqm/+bug/442934 too .  He has it in his to-do list to pursue.
<ubottu> Launchpad bug 442934 in pqm "unknown url type 'bzr+ssh://bazaar.launchpad.net/$user/$project/$branch;revno=$int'" [Undecided,New]
<matsubara> Chex, about the splitit setup, are we expecting any new oops ids? I glanced over the email about it and notice that there will be lots of new instances running
<matsubara> s/oops ids/oops prefixes/
<gary_poster> (I'm interested too, but he's going to do something about it :-) )
<allenap> Chex: Was that for lp-source-dependencies?
<stub> If I recall from the email conversation, that bug is on a feature we don't actually need to be using.
<stub> (because we have no need to freeze the revision of the dependencies branch being used).
<allenap> Chex: Because... what stub said.
<matsubara> rockstar, any news about the code browse issue?
<Chex> matsubara: not sure about the OOPS ids, but I believe we will need that. I will check on that for you
<rockstar> matsubara, not yet.
<matsubara> [action] chex to check the new oops prefixes added by the splitit setup and inform matsubara
<MootBot> ACTION received:  chex to check the new oops prefixes added by the splitit setup and inform matsubara
<mthaddon> matsubara: for new OOPS ids see the lp-production-configs trunk branch
<Chex> allenap: ok, I will check into that bug status, then. I am not sure of that issue myself.
<gary_poster> stub, allenap, Chex: we need bug 442934 for the branches in sourcecode
<ubottu> Launchpad bug 442934 in pqm "unknown url type 'bzr+ssh://bazaar.launchpad.net/$user/$project/$branch;revno=$int'" [Undecided,New] https://launchpad.net/bugs/442934
<matsubara> thanks mthaddon 
<mthaddon> matsubara: can we 'unaction' that one from Chex?
<gary_poster> not for the buildout-related branch
<matsubara> [action] matsubara to ignore the previous Chex action and look in lp-production-configs for the new oops prefixes.
<MootBot> ACTION received:  matsubara to ignore the previous Chex action and look in lp-production-configs for the new oops prefixes.
<mthaddon> cool, thx
<matsubara> Chex, about the QA info column, do you have a handy link?
<Chex> matsubara: sure, I should have included that
<Chex> https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
<matsubara> thanks Chex 
<matsubara> [action] all QA contacts to inform their teams about the new QA column and what they should do about it.
<MootBot> ACTION received:  all QA contacts to inform their teams about the new QA column and what they should do about it.
<matsubara> Chex, did you send an email about the new QA column to the list?
<matsubara> if not, it'd be great if you could send it and explain what you the losas expects there
<Chex> matsubara: no I did not, but I can do that for you.
<matsubara> thanks Chex 
<Chex> matsubara: sure, no problem
<matsubara> [action] Chex to email the list about the new QA column in https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
<MootBot> ACTION received:  Chex to email the list about the new QA column in https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
<matsubara> let's move on
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<stub> The database patch approval process has changed. Previously, they went through me and then had to get a second level of approval by Mark at a monthly meeting. Now they just need to be approved by both myself and jml. We should now be able to track everything with merge proposals and the system will be much less time sensitive, allowing reviews to happen throughout the cycle.
<stub> A new database replica is being brought online as I type. It will be left running as a replica for a few days to ensure it has no problems keeping up (it doesn't have much RAM). Assuming the burn in goes fine, it will become the master for the authdb replication set as part of the ongoing process to separate the authentication systems from Launchpad.
<stub> The databases will all need to be bounced soon to increase the maximum number of connections. We need to do this as we are bringing new appservers on line (per the results of the performance sprint in Montreal). Assuming tests on staging go well, we should be able to do this with no end-user noticeable effects as the database reconnection handling with handle the short outage just fine.
<stub> Bug #426823 is tracking the progress of getting Launchpad to run with PostgreSQL 8.4. At the moment, it looks like we will need to wait until 8.4.2 as we are tickling a PostgreSQL bug. This bug has been tracked down with upstream and a patch is available, but I'd prefer if we stuck to official releases than assembling our own package. We are also blocked by the release of Slony-I 1.2.17, assuming we don't want to move forward with using th
<ubottu> Launchpad bug 426823 in launchpad-foundations "Update to PostgreSQL 8.4" [Medium,In progress] https://launchpad.net/bugs/426823
<stub> There are still some test failures to address with PG 8.4. Some are ordering problems where we were incorrectly relying on database ordering, other issues might be more serious where we are obtaining different results than under 8.3. I may pass some of these onto the teams to see if something is broken under 8.4 or if the issue is benign.
<stub> oot.
<matsubara> questions for stub?
<matsubara> thanks stub 
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no new proposed item
<matsubara> anything else before I close?
<matsubara> 4
<matsubara> 3
<matsubara> 2
<matsubara> 1
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:36.
<gary_poster> thanks matsubara!
<allenap> stub: I'm just catching up. One line got truncated, ends with "assuming we don't want to move forward with using th". Can you paste from there?
<stub> assuming we don't want to move forward with using the release candidate. I'll land the branch allowing people to run 'make schema' with PG 8.4, but I don't recommend anyone do this yet because the PG bug we are tickling is fairly pervasive and it will end in frustration.