## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
* danilos has quit (Nick collision from services.)
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<matsubara> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us! 
<sinzui> me
<al-maisan> me
<danilo_> me
<mrjazzcat> me
<mbarnett> me
<matsubara> sorry mrjazzcat, I always forget to ping you about the meeting. I'll add you to the "Who should be here?" section if you don't mind
<mrjazzcat> yes, please
<matsubara> on the MeetingAgenda page, I mean
<mrjazzcat> no worries
<matsubara> [action] add brian to the list of attendees in the MeetingAgenda page
<MootBot> ACTION received:  add brian to the list of attendees in the MeetingAgenda page
<matsubara> Ursula won't be around today
<matsubara> and I'll be standing in for Gary
<matsubara> rockstar, hi, around?
<matsubara> allenap, hi
<matsubara> well, let's move on and then Gavin and Paul can join in later
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara>  * DBA report (stub)
<matsubara>  * Proposed items
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * allenap to dig the master bug of OOPS-1474EA771
<matsubara>  * salgado to take a look in the TypeError oopses (OOPS-1479S1000)
<matsubara>    * already did that, this is bug 403281, it happened because mthaddon was testing the new read-only switch on staging.
<matsubara>  * rockstar to take a look in OOPS-1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1479S1000
<ubottu> Launchpad bug 403281 in launchpad-foundations "public xmlrpc requests broken during read only period" [Undecided,Triaged] https://launchpad.net/bugs/403281
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<matsubara> ok, so I'll re-add both items for allenap and rockstar 
<matsubara> [action] * allenap to dig the master bug of OOPS-1474EA771
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<MootBot> ACTION received:  * allenap to dig the master bug of OOPS-1474EA771
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<matsubara> [action] * rockstar to take a look in OOPS-1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<MootBot> ACTION received:  * rockstar to take a look in OOPS-1480CMP1
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1480CMP1
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> we have some oops reports but most of them foundations issues
<matsubara> https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
<matsubara> Looks like an anonymous user is trying to do some operation which (s)he's not allowed. Should we really log an oops for this?
<matsubara> maybe related to https://bugs.edge.launchpad.net/launchpad-foundations/+bug/271029
<matsubara> More non-informational disconnectionerrors https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489J147
<matsubara> InternalError after ther rollout https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489C1094
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489J147
<ubottu> Ubuntu bug 271029 in launchpad-foundations "ForbiddenAttribute exception raised changing property of object" [Medium,Triaged]
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489C1094
<matsubara> code team, BranchMergeProposalExists https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
<matsubara> so, that's it and there's no one from Code to take a look at the BranchMergeProposalExists one
<matsubara> [action] matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
<MootBot> ACTION received:  matsubara to email Tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA174
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA174
<matsubara> [action] matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
<MootBot> ACTION received:  matsubara to talk to leonard about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1488EA884
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1488EA884
<matsubara> [action] matsubara to talk to salgado about More non-informational disconnectionerrors https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489J147
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489J147
<MootBot> ACTION received:  matsubara to talk to salgado about More non-informational disconnectionerrors https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489J147
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489J147
<matsubara> [action] matsubara to talk to stub or gary about InternalError after ther rollout https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489C1094
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489C1094
<MootBot> ACTION received:  matsubara to talk to stub or gary about InternalError after ther rollout https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1489C1094
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1489C1094
<matsubara> lovely, looks like I'm running the meeting all by myself heheh
<rockstar> me
<allenap> me
* noodles775 has quit (Read error: 110 (Connection timed out))
<al-maisan> :)
<matsubara> on the broken scripts side
<matsubara> sinzui, Scripts failed to run: loganberry:send-person-notifications seems to be broken
<matsubara> sinzui, could you take a look and reply to the list?
* noodles775 (n=miken@canonical/launchpad/noodles775) has joined #launchpad-meeting
<sinzui> matsubara: all scripts appear to be broken
<matsubara> all?
<sinzui> They are not running and I am tempted to say something new was added that is taking forever and a day
<matsubara> I only see notifications for send-person-notifications and garbo-hourly
<matsubara> sinzui, can you confirm and reply to the list that's the case, at least for the send-person-notifications one?
<matsubara> I'll ask losas and/or stub about garbo-hourly not running as well
<allenap> matsubara: Re. OOPS-1474EA771, it's bug 508302, and deryck is working on it today.
<ubottu> Launchpad bug 508302 in malone "NotImplementedError OOPS when reporting a bug" [High,In progress] https://launchpad.net/bugs/508302
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<matsubara> thanks allenap, I'll adjust the bug link on that oops report
<matsubara> [action] matsubara to fix bug link on OOPS-1474EA771 to point to bug 508302
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<MootBot> ACTION received:  matsubara to fix bug link on OOPS-1474EA771 to point to bug 508302
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1474EA771
<matsubara> [action] sinzui to investigate failure on send-person-notifications and reply to the list with his findings
<MootBot> ACTION received:  sinzui to investigate failure on send-person-notifications and reply to the list with his findings
<matsubara> btw, updatebranches script also failed recently but that's been fixed by spm. the new rollout changed the script name and losas updated the notification thing to recognize the new name
<matsubara> on the critical bugs side
<rockstar> matsubara, updatebranches no longer runs.
<matsubara> we have 3 critical bugs
<mthaddon> matsubara: er, not quite
<rockstar> It's been replaced by scan_branches
<mthaddon> matsubara: we've had to revert it a bunch of times
<matsubara> mthaddon, hmm no? spm's email seems to indicate that
<mthaddon> matsubara: spm went to bed a while ago - a new problem was discovered since then
<mthaddon> matsubara: abentley and Chex have been working on it
<matsubara> oh, I was looking at this latest email to the list replying to one of the script failures notification
<matsubara> well, if they're already working on it, it's ok. :-)
<mthaddon> matsubara: not really...
<matsubara> mthaddon, no? what else is expected?
<mthaddon> matsubara: as I understand it, we've reverted to the old script because we still don't know what was wrong
<mthaddon> matsubara: and the fact that we've reverted between the old and new scripts twice now on production is a problem in itself
* salgado (n=salgado@canonical/launchpad/salgado) has joined #launchpad-meeting
<matsubara> mthaddon, I meant it's ok in the sense that people are already working on a solution and there's nothing much to be done during this meeting to have people act on it
<mthaddon> matsubara: and also the fact that the first we heard about the problem was from a user report
<mthaddon> i.e. we don't have a good measure of when this problem is even happening
<mthaddon> matsubara: maybe not, but I'd like a bit of discussion about this class of problem and what can be done to prevent it in the future
<danilo_> mthaddon: what is the exact problem that we need to be able to track? (sorry, I am not fully up to date on what broke)
<mthaddon> danilo_: aiui email notifications of branch updates failed to be sent out
<gary_poster> mthaddon: "reverted ...twice...on production": I think we all agree this sucks.  However, AIUI, this was successfully QAd.  Either the QA was bad, or staging is not close enough to prod in some way.  I don't think we know yet.
<matsubara> mthaddon, I'm unaware of the details as well. My expectation is that a IncidentLog will be filed and action to prevent it will be included in the incident log
<danilo_> mthaddon: ah, right, that could have a bigger impact (it might be harming us in translations as well)
<mthaddon> matsubara: this doesn't really qualify as an incident log item since there's no measurable service that's been interrupted (we don't have any kind of nagios monitoring of this) - I guess I'm asking how we plan to approach it from here
<mthaddon> and how we got into this situation
<danilo_> mthaddon, gary_poster, matsubara: we are obviously missing a dedicated "communications person" for this specific item (someone to keep the entire situation in check); we've discussed that approach before, it'd be nice to find someone who can offload the communication side from abentley and others working on it
<gary_poster> danilo_: to the degree there's a failure there (communications), it'd probably be mine as RM
<gary_poster> maybe we can have somebody else too
<gary_poster> but that's RM stuff
<gary_poster> but AIUI that's not the prob
<danilo_> gary_poster, not necessarily, we discussed this in a TL call a few weeks (months?) back where we need someone to communicate with everyone
<gary_poster> maybe so
<gary_poster> but probs I see:
<danilo_> gary_poster, it's mostly about having someone take responsibility for making sure problems are visible and we know what's going on
<gary_poster> - we didn't catch this on staging.  Why?
<gary_poster> either QA was bad or staging is too diff
<gary_poster> we need to know why
<gary_poster> and fix it
<mthaddon> yep, I agree with that
<gary_poster> then also, unless I misunderstand, mthaddon is saying that we don't have an automated nagios-like process verifying basic success on production for this thing
<danilo_> gary_poster, neither of those is easy to fix (one depends on people always DTRT, another on machines always DTRT), so we need to be able to easily find out when it's broken rather than wait for users to report it
<gary_poster> danilo_: but doesn't that depend on one of the three things I said?  (people DTRT, machines DTRT, nagios-like-thing DTRT)
<danilo_> gary_poster, it does, I was typing before you typed the last one :)
<mthaddon> gary_poster: it's possible we can't do that for *everything*, but if we decide this is a sufficiently important thing that we care about it if it fails, it sounds like we need to monitor it somehow, yeah (possibly we are already with OOPSes, but why didn't we catch it til a user told us about it?)
<gary_poster> :-) ok
<danilo_> gary_poster, the 4th is lack of coordination and communication :)
* salgado-lunch has quit (Read error: 110 (Connection timed out))
<gary_poster> mthaddon: right. For me, this gets to my "too many different kinds of moving parts" in our architecture. If we have fewer moving parts then we can institute more uniform nagios-like-checks.
<gary_poster> maybe the jobs system can help with this
<danilo_> anyway, gary_poster, I think we should just raise the importance of ensuring sufficient monitoring of this part of code-hosting by thumper, and we can be done with the topic
<gary_poster> maybe we can architect the jobs system to give us a nagios-like hook
<danilo_> gary_poster, we don't have to solve the problem here :)
<gary_poster> because doing it with cron scripts is a one-per job
<matsubara> danilo_, can you raise the topic in the next TL meeting?
<gary_poster> danilo_: ack.  I kind of disagree with your summary though, and your action item, so that's why I'm continuing to blather :-)
<danilo_> matsubara, we are having a week long TL meeting next week, so it'd be best to action it for someone from code team to pass it on to thumper, imho :)
<gary_poster> (IOW, this is not a problem for thumper, it is a problem for Björn, team leads, etc.)
<danilo_> gary_poster, well, sure, I agree, but one step at a time
<gary_poster> matsubara: two action items: :-)
<matsubara> [action] rockstar to raise the importance of ensuring sufficient monitoring of this part (i.e. branch updates emails failing to be delivered) of code-hosting by thumper
<MootBot> ACTION received:  rockstar to raise the importance of ensuring sufficient monitoring of this part (i.e. branch updates emails failing to be delivered) of code-hosting by thumper
<danilo_> gary_poster, there's immediate problem and then there's the elegant solution; I'm always for fixing the immediate problem first and having the elegant solution come out of that
<gary_poster> yeah, that's number one
<gary_poster> number two is gary to bring up archtecture concerns to team lead mtg :-)
<danilo_> gary_poster, as for the other one, I think it ties in well with what we discussed today and what we'll want to discuss anyway
<matsubara> [action] TLs + Bjorn to talk about "too many different kinds of moving parts" in our architecture. If we have fewer moving parts then we can institute more uniform nagios-like-checks.
<MootBot> ACTION received:  TLs + Bjorn to talk about "too many different kinds of moving parts" in our architecture. If we have fewer moving parts then we can institute more uniform nagios-like-checks.
<matsubara> does that summarize it well?
<gary_poster> yeah thank you.  though it's probably my action, since I'm the one with the bee in my bonnet :-)  but that's fine
<danilo_> gary_poster, matsubara: I don't like action items like that because they put no responsibility on anyone in particular, thus meaning that if they get done, they get done unrelated to the action item; thus, you don't really need it
<gary_poster> so give it to me :-)
<matsubara> danilo_, I'll add it to gary's queue when I add the summary to the MeetingAgenda page
<danilo_> gary_poster, heh, that's ok, I am certain we would have discussed this regardless of us having any particular action item
<danilo_> matsubara, sure, thanks
<gary_poster> :-)
<matsubara> it serves as a reminder as well
<matsubara> anyway, thanks for the comments
<rockstar> It fairness, the "not getting branch update emails" thing was because a rather large part of the code hosting system was made into a job.
<gary_poster> To whom are you being fair? :-)
<gary_poster> Never mind, I'll be quiet :-)
<danilo_> :)
<matsubara> we have 3 critical bugs, one in progress, one fix committed
<rockstar> I'm not sure how "sufficient monitoring" would have fixed this.
<matsubara> the other one is triaged, bug 511567
<ubottu> Launchpad bug 511567 in launchpad-foundations "Can't remove authorised app" [Critical,Triaged] https://launchpad.net/bugs/511567
<rockstar> gary_poster, to the code team in general.
<matsubara> hmm
<matsubara> that's a dupe
<danilo_> rockstar, sufficient monitoring of scripts that do this
<matsubara> and I filed that bug a few days ago
<rockstar> danilo_, howso?
<matsubara> or maybe I filed the dupe
<gary_poster> rockstar: ah, gotcha.  Tim can beat us into shape at the TL sprint so we understand.
<rockstar> gary_poster, yeah, I'll talk to him.
<gary_poster> cool
<danilo_> rockstar, monitoring should have caught the problem (i.e. "hey, this script is failing"); I won't pretend to understand the entire problem, so we might be entirely off base, but we should be able to check our service level
<rockstar> danilo_, there wasn't a script failing.
<rockstar> It ran fine, it was just a new script that had apparently left out some old functionality.
<danilo_> rockstar, right, never mind the "implementation details", the problem is: "why we didn't catch it before someone told us it's failing"; there's not necessarily a technical solution
<danilo_> matsubara, am I still on the channel?
<matsubara> yes
<danilo_> oh, ok, it's just everybody being quite :)
<danilo_> matsubara, I think we should go on
<matsubara> sorry, I was looking for a bug report to dupe against 511567
<matsubara> anyway
<matsubara> thanks
<matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
<MootBot> New Topic:  * Operations report (mthaddon/Chex/spm/mbarnett)
<matsubara> hello?
<matsubara> Chex, mbarnett ?
<mbarnett> sorry
<Chex> sorry
<Chex> here is the report
<Chex> - LP rollout 10.01 Wednesday was successful:
<Chex>     : See https://wiki.canonical.com/InformationInfrastructure/OSA/LPRollout20100127 for more details.
<Chex>     : The read-only switch left idle connections to the master DB, it is currently being investigated
<Chex> - New LP Appserver is online, some issues with internal access, but now everything is OK.
<Chex> - New branch-scanner having issues, just reverted back to old again.  Based on meeting dicsussion here,
<Chex>         continuing to address.
<Chex> and thats all for us.  Any questions/comments?
<matsubara> Chex, what's this new LP appserver online? I guess I'll have to tell oops-tools about oops reports from it?
<matsubara> [action] matsubara to update oops-tools to know about the new lp appserver
<MootBot> ACTION received:  matsubara to update oops-tools to know about the new lp appserver
<noodles775> Chex: do you know if the new servers have access to the private librarian?
<mbarnett> matsubara: soybean was recently put online as a replacement for gangotri +
<mbarnett> noodles775: that was resolved earlier today
<matsubara> mbarnett, oh, so it's using the same config files?
<noodles775> A user was seeing about 1 in 4 requests to download a... ah, great, thanks!
<mbarnett> matsubara: it took over lpnet1, lpnet2, and edge1 from gangotri, stole lpnet9 from gandwana, and added a sparkly new lpnet15 standard lpnet appserver
<matsubara> mbarnett, ok, it's the new lpnet15 instance I care about. I'll check the configs and update oops-tools accordingly
<matsubara> thanks
<matsubara> moving on
<mbarnett> matsubara: thank you.
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<matsubara> stub sent the report to the list
<matsubara> allenap, he mentioned something about checkwatches being very cpu intensive. it's probably of interest of the Bugs team
<allenap> mars: deryck has just forwarded the message to me.
<allenap> matsubara: ^
<matsubara> thanks allenap 
<matsubara> [TOPIC] * Proposed items
<MootBot> New Topic:  * Proposed items
<matsubara> no proposed items
<matsubara> which brings this meeting to a close
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs. 
<matsubara> and sorry for the delay
<matsubara> #endmeeting