#format IRC
<Ursinha> #startmeeting
<Ursinha> Welcome to this week's Launchpad Production Meeting.  For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
<Ursinha> [TOPIC] Roll Call
<Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
<MootBot> Meeting started at 09:02. The chair is Ursinha.
<MootBot> New Topic:  Roll Call
<Ursinha> hi MootBot 
<Ursinha> me
<herb> me
<bac> me
<intellectronica> me
<gary_poster> me
<Ursinha> rockstar, ping
<cprov> me
<Ursinha> henninge is mia
<rockstar> me (barely)
<Ursinha> :)
<Ursinha> well, no issues for translations, so no much of a problem
<Ursinha> alright
<Ursinha> [TOPIC] Agenda
<Ursinha>  * Actions from last meeting
<Ursinha>  * Oops report & Critical Bugs-
<Ursinha>  * Operations report (mthaddon/herb/spm)
<Ursinha>  * DBA report (DBA contact)
<MootBot> New Topic:  Agenda
<Ursinha> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<Ursinha>  1) matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time
<Ursinha>  2) TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week (26th to 30th Jan)
<Ursinha>  3) matsubara to file bugs for OOPS-1116EC265 (bug 320293), OOPS-1113EB6 (bug 276144)
<ubottu> Launchpad bug 320293 in launchpad-bazaar "OOPS traversing to branch on person page" [Medium,Confirmed] https://launchpad.net/bugs/320293
<ubottu> Launchpad bug 276144 in launchpad-bazaar "OOPS when trying to delete stacked-on branches." [Medium,Triaged] https://launchpad.net/bugs/276144
<Ursinha> 1 - I did that. From now on to discuss meeting time changes just add it to the list of meeting topics, on MeetingAgenda page
<Ursinha> 2 - okay as well
<Ursinha> next!
<Ursinha> [TOPIC] * Oops report & Critical Bugs
<MootBot> New Topic:  * Oops report & Critical Bugs
<Ursinha> three oopses and one bug
<Ursinha> the oopses
<Ursinha> MissingMergeDirective: OOPS-1125CEMAIL1
<Ursinha> this is yours rockstar?
* Ursinha kicks ubottu 
<rockstar> Ursinha, I can look at it
<Ursinha> rockstar, thanks 
<Ursinha> bac, to registry: OOPS-1125A165, OOPS-1125A1096 - this second one is the same in the email sinzui sent to barry
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
* henninge (n=henning@i577A8E79.versanet.de) has joined #launchpad-meeting
<henninge> me!
<Ursinha> hi henninge!
<henninge> Ursinha: hi! :)
<bac> Ursinha: barry is working on the second one
<henninge> sorry for the delay ...
<Ursinha> bac, do you know if there's an open bug for that?
* mars has quit (Remote closed the connection)
<Ursinha> intellectronica, bug 279561, that's affecting people more frequently lately
<ubottu> Launchpad bug 279561 in malone "No url for <Message at ...> when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561
* intellectronica shoegazing
<bac> Ursinha: i don't.  i'll check with barry when he returns from afk
<intellectronica> sorry, it's still a mystery
<intellectronica> i'll go back to working on it now
<Ursinha> intellectronica, do you have time to deal with that? it's causing a lot of oopses and annoying people
<Ursinha> bac, thanks 
<intellectronica> Ursinha: i do now
<Ursinha> [action] bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
<MootBot> ACTION received:  bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
<Ursinha> bac, about the other one, do you have any clues?
<Ursinha> intellectronica, thanks
<Ursinha> [action] intellectronica to work on bug 279561
<MootBot> ACTION received:  intellectronica to work on bug 279561
<ubottu> Launchpad bug 279561 in malone "No url for <Message at ...> when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561
<Ursinha> [action] rockstar to check OOPS-1125CEMAIL1
<MootBot> ACTION received:  rockstar to check OOPS-1125CEMAIL1
<Ursinha> any other issues you may want to point?
<bac> Ursinha: no clue ATM.  will check it out
<Ursinha> thanks bac
<Ursinha> I appreciate that
<Ursinha> [action] bac to take a look at OOPS-1125A165
<MootBot> ACTION received:  bac to take a look at OOPS-1125A165
<Ursinha> well, moving on
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
<ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
<Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> * 2008-01-28 - Yesterday - We rolled out 2.2.1. Total downtime was approximately 35 minutes.
<herb> * Even after the rollout, we continue to have problems as a result of bug #156453 and bug #118625. I cannont emphasize enough how much of a pain point this is becoming for the LOSAs. More than a quarter of the incidents that we've had to deal with in the last year have been related to these two bugs.
<ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
<ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
<herb> * We had a couple of incidents in the last week relating to bug #260171.
<ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
<herb> * Do we expect to roll out any additional patches this week? If so, when?
<Ursinha> herb, I think at least one, I'll talk to kiko and confirm that to you
<herb> ok
<herb> Ursinha: do you suspect it will be rolled out today or tomorrow?
<henninge> Ursinha: He approved one of mine, too.
<Ursinha> [action] Ursinha to check with kiko if any other rollouts will happen this week
<MootBot> ACTION received:  Ursinha to check with kiko if any other rollouts will happen this week
<herb> I can wait, I was just curious if you have a best guess.
<Ursinha> herb, guess tomorrow, but it depends on if we're having more issues to be fixed
<herb> ok
<herb> thanks
<Ursinha> henninge, cool
<Ursinha> thanks herb 
<Ursinha> I took note of the "pain to the LOSAs" about that two bugs
<herb> Ursinha: thanks. much appreciated
<Ursinha> well, next
<Ursinha> [TOPIC] * DBA report (DBA contact)
<Ursinha> stub sent one email about it:
<Ursinha> Looks like the upgrade went smoothly and nothing exciting is happening. Yay.
<Ursinha> Although neither db server is particularly heavily loaded on average, we do see spikes sometimes. Also, the slave is still more loaded than production. As I don't expect us to get a second slave in the near future, I'm thinking we need to look into some better load balancing to move some of the slave load back to the master.
<Ursinha> Config changes on one or two of the appserver instances to use the master as the slave would work in the short term. Long term I think we could be better with a load balancer like pgbouncer or pgpool (1 or 2).
<MootBot> New Topic:  * DBA report (DBA contact)
<Ursinha> that's all from stub
<Ursinha> anything else?
<Ursinha> anybody wants to point or discuss something?
<Ursinha> 5
<Ursinha> 4
<Ursinha> 3
<Ursinha> 2
<Ursinha> 1
<Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
<Ursinha> #endmeeting