## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<rockstar> ni!
* herb__ has quit (Client Quit)
<danilos> me
<adeuring> me
<adeuring> (allenap is sick)
<danilos> (or "coo", if anyone knows about kin dza dza ;)
<matsubara> gary_poster, Chex, bigjools: hi
<mbarnett> hello
<matsubara> sinzui, hi
<gary_poster> me
<gary_poster> and hi
<matsubara> :-)
<sinzui> me
<mbarnett> me
<matsubara> apologies from Stuart and Ursula
<bigjools> me
<Chex> hello
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara> * matsubara to trawl logs related to high load on edge on 2009-09-09 ~1830UTC and ping Chex about it
<matsubara> * matsubara to email the devel list about the new ErrorReportingUtility method
<matsubara>     * done
<matsubara> * matsubara to file a bug to have the HWSubmissionMissingFields oopses as informational only (note to self: see bug 438671 for more details)
<matsubara>     * filed https://bugs.edge.launchpad.net/malone/+bug/446660
<matsubara> * matsubara to look in lp-production-configs for the new oops prefixes.
<matsubara> * all QA contacts to inform their teams about the new QA column and what they should do about it.
<matsubara> * Chex to email the list about the new QA column in https://wiki.canonical.com/InformationInfrastructure/OSA/LPIncidentLog
<ubottu> Launchpad bug 438671 in checkbox "HWSubmissionMissingFields OOPS on +hwdb/+submit" [Undecided,Confirmed] https://launchpad.net/bugs/438671
<ubottu> Launchpad bug 446660 in malone "HWSubmissionMissingFields exceptions should be updated to be informational only" [High,Triaged]
<matsubara> I still haven't checked the high load logs. Chex or mthaddon, did you notice high loads after the 2009-09-09? 
<Chex> matsubara: we still have been seeing some high loads, yes
<matsubara> I did look the new prefixes on lp-productions-configs. I need to update oops-tools to recognize those
<matsubara> I also noticed that some oops prefixes will conflict with existing ones, so I need to sort that out with...
<matsubara> losas I guess
<matsubara> [action] matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas
<MootBot> ACTION received:  matsubara to file a bug on oops-tools to recognize new oops prefixes and sort out conflicting prefixes with losas
<matsubara> Chex, re: the high load, could you take on the task of analysing the logs? my idea was to correlate information from the app servers logs with the apache logs and see if that could shed some light.
<matsubara> mthaddon emailed the list about the new QA column, so everyone, read it and spread the word to your teams, please.
<Chex> Chex: yes sure, I can look at that.
<danilos> matsubara: it has just been discussed in the TL call as well, flacoste will champion the process
<Chex> matsubara: ^^  I mean..
<danilos> matsubara: (about QA Info column on LP incident log)
<matsubara> Chex, cool, thanks a lot. ping me if you need any info on that
<matsubara> danilos, cool. thanks!
<matsubara> [action] Chex to check app server logs and apache logs to see if it can shed any light in the high load issue.
<MootBot> ACTION received:  Chex to check app server logs and apache logs to see if it can shed any light in the high load issue.
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> we're seeing a bunch on DisconnectionErrors which are not informational only
<matsubara> which means, the Retry mechanism is not enough for those cases.
<gary_poster> matsubara: are these the ones on the xmlrpc server?
<gary_poster> or something else?
<matsubara> gary_poster, yes, most of them on xmlrpc server
<matsubara> but there are a few, like OOPS-1383I246, in login.launchpad.net
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1383I246
<gary_poster> matsubara: right.  I investigated and could not duplicate the ones in the xmlrpc server.  Kicking the xmlrpc server made them go away.  There's a bug number which I can get in a moment.  After discussing with flacoste, I think the best we can hope for is to figure out a way to add more diagnostic information should the problem happen again
<matsubara> gary_poster, ok, I take this is a foundations taks then. let me know the bug number please (or I'll file a new one for the more diagnostic info needed issue, if that's not what the bug you mentioned is about)
<gary_poster> bug 450593 .   Stuart has a follow up: check with losas if there were any unusual activity ATM
<ubottu> Launchpad bug 450593 in launchpad-foundations "Lots of DisconnectionErrors on xmlrpc server - staging" [Undecided,New] https://launchpad.net/bugs/450593
<gary_poster> I think a comment saying that we should address by adding diagnostic information in case there is a repeat would be sufficient.  I'll do that.
<matsubara> thanks gary_poster 
<matsubara> apart from that we have a bunch of oopses that will need fixing given the new zero oops policy.
<matsubara> Ursula will keep an eye on those for now and let the teams lead which ones are happening more frequently
<matsubara> we had some script failures last week
<matsubara> the main one seems to be the branch-puller which was already discussed in the list
<matsubara> checkwatches failed on the 13th, but since no other email came out, I assume it was a blip. adeuring, can you confirm?
<adeuring> matsubara: erm, I have no idea...
<matsubara> and the product-release finder and update-cache failed to run on the 14th
<adeuring> matsubara: I'll ask Graham
<matsubara> sinzui, do you know what's up with the product release finder script?
<matsubara> who's owns the update-cache script? 
<matsubara> s/'s//
<matsubara> thanks adeuring 
<gary_poster> I don't know; looking
<matsubara> [action] adeuring to check with gmb about checkwatches failure
<MootBot> ACTION received:  adeuring to check with gmb about checkwatches failure
<sinzui> matsubara: No, but I think the issue is not that it failed, bu that a long process prevented it from running
<matsubara> sinzui, right, that'd explain. could you check that's the root cause and reply to the list?
<sinzui> matsubara: okay
<matsubara> maybe the update-cache failure happened for the same reason
<gary_poster> matsubara: I don't see an update-cache script in the LP tree.  (I do see variants like update-download-cache)
<matsubara> just a reminder to everyone, if a script fails and your team owns that script, please reply to the failure email saying that someone is taking a look at it.
<matsubara> gary_poster, all I see is: "The script 'update-cache' didn't run on 'loganberry' between 2009-10-14 04:00:08 and 2009-10-14 22:00:08 (last seen 2009-10-13 11:36:51.345188)" not sure which script that one is monitoring.
<matsubara> for the critical bugs section, we have 4 bugs, 3 fix committed and 1 in progress
<matsubara> danilos, the one in progress is assigned to henning but he's on vacation
<matsubara> is it really critical?
<danilos> matsubara: I'd have to check, sorry for not being on top of this
<gary_poster> (I also looked for update-cache in lp-production-configs.  not there either.)
<matsubara> gary_poster, I think it's cronscripts/update-pkgcache.py. IIRC, the losas script monitoring tool uses the script name defined in LaunchpadCronScript
<matsubara> [action] danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance
<MootBot> ACTION received:  danilos to check bug 438039, assess if it's really critical. if it's is, land a fix, if it's not, update the importance
<ubottu> Launchpad bug 438039 in rosetta "bzr branch import script oopses sometimes" [Critical,In progress] https://launchpad.net/bugs/438039
<gary_poster> matsubara: oh ok, thanks.  that script is either the one salgado was talking about that he owns, or something for soyuz, seems to me.
<bigjools> it's traditionally maintained by soyuz
<gary_poster> bigjools: ok, thanks
<bigjools> but in the new world order it could be registry
<matsubara> bigjools, can you confirm that update-cache failure described in the "Subject: Scripts failed to run: loganberry:productreleasefinder, loganberry:update-cache" refers to the update-pckg.py and reply back to the email sent to the list?
<matsubara> ok, you just did :-)
<bigjools> it doesn't look like update-packagecache
<bigjools> errr ah it is
<matsubara> bigjools, it's the only script that has update-cache string in cronscripts/
<bigjools> sorry got confused by seeing productreleasefinder
<matsubara> [action] bigjools to investigate update-cache failure and reply back to the list
<MootBot> ACTION received:  bigjools to investigate update-cache failure and reply back to the list
<matsubara> bigjools, you might want to coordinate with sinzui since he'll check the product release failure one and suspects it might have failed because of a long running process
<bigjools> matsubara: is there an oops?
<sinzui> only an email that it did not run
<matsubara> bigjools, nope
<bigjools> and it was a one-off?
<sinzui> bigjools: it did not start, and that is 99% of the time the fault of a long running process
<bigjools> ok
* sinzui really does not think about the issue until it happens two days in a row
<bigjools> and me
<matsubara> sinzui, perhaps the script monitoring should have such a feature
<matsubara> but anyway, sorry for taking so long on this section
<matsubara> thanks eveyrone
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<Chex> hello everyone
<gary_poster> hi
<Chex> sorry, notes failure:
<Chex> - LP Ship-it progress:
<Chex> ; LP shipit is live on the new servers
<Chex> ; Nigel Pugh is now in charge of approving CPs to those servers
<Chex> ; We are still working on the new front-ends for LP Login and LP itself
<Chex> - Buildd-manager DB restart issue/bugs: Bugs 451351 & 451349 have been
<ubottu> Launchpad bug 451351 in soyuz "buildd-manager doesn't give us a good way of determining it's in a failed state" [High,Triaged] https://launchpad.net/bugs/451351
<Chex> filed to address this issue, any movement to fix this problem?
<Chex> - QA column in Incident Log: Tom sent a email to LP list on Oct 12, has
<Chex> anyone reviewed the email and have comments/concerns about it?
<matsubara> Chex, are oops reports from those new servers going to be rsync'ed to devpad? such oopses are supposed to be included in LP oops summaries?
<Chex> LP Incidents of note: ; Applied: CP 9660 to lpnet, CP 9679 to lpnet
<Chex>     ; Small LP outage (8 mins) : App servers (and
<Chex>          librarians) didn't reconnect & had to be restarted after LP DBs
<Chex> were restarted: Bug filed: 451093
<Chex> and thats our report for this week. sorry for the troubles there
<bigjools> Chex: I am looking into 451351 but don't expect anything soon, it's a hard problem
* noodles775 has quit ("Leaving")
<Chex> matsubara: I am not sure on the status of oops summaries on the new servers, I will check on that
<matsubara> Chex, cool, thanks.
<Chex> bigjools: ok, thanks, just looking for status of progress.
<matsubara> Chex, danilos mentioned that QA column things was discussed today in the TL meeting and flacoste will champion the process.
* andrea-bs has quit (Remote closed the connection)
<Chex> matsubara: ok, that is great to hear.
<matsubara> Chex, thanks for the report
<matsubara> let me move on as we are overdue
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> The new replica to become the master for the authentication service has been taken offline, as the hardware was showing signs of strain keeping up with Launchpad's write load. The hardware is being beefed up to cope. The alternative is to just put the authdb replication set on this server and have the authentication service appservers connect to the main launchpad databases for the data they need to pull from the lpmain repl
<matsubara> ication set.
<matsubara> Nothing else to report.
<matsubara> that came from Stuart. any questions about dba's report?
<matsubara> ok, I'll take that as a no :-)
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no new proposed items
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> sorry for overrunning 
<matsubara> #endmeeting
<MootBot> Meeting finished at 10:51.