1 <Ursinha> #startmeeting
   2 <Ursinha> Welcome to this week's Launchpad Production Meeting.  For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   3 <Ursinha> [TOPIC] Roll Call
   4 <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
   5 <MootBot> Meeting started at 09:02. The chair is Ursinha.
   6 <MootBot> New Topic:  Roll Call
   7 <Ursinha> hi MootBot 
   8 <Ursinha> me
   9 <herb> me
  10 <bac> me
  11 <intellectronica> me
  12 <gary_poster> me
  13 <Ursinha> rockstar, ping
  14 <cprov> me
  15 <Ursinha> henninge is mia
  16 <rockstar> me (barely)
  17 <Ursinha> :)
  18 <Ursinha> well, no issues for translations, so no much of a problem
  19 <Ursinha> alright
  20 <Ursinha> [TOPIC] Agenda
  21 <Ursinha>  * Actions from last meeting
  22 <Ursinha>  * Oops report & Critical Bugs-
  23 <Ursinha>  * Operations report (mthaddon/herb/spm)
  24 <Ursinha>  * DBA report (DBA contact)
  25 <MootBot> New Topic:  Agenda
  26 <Ursinha> [TOPIC] * Actions from last meeting
  27 <MootBot> New Topic:  * Actions from last meeting
  28 <Ursinha>  1) matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time
  29 <Ursinha>  2) TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week (26th to 30th Jan)
  30 <Ursinha>  3) matsubara to file bugs for OOPS-1116EC265 (bug 320293), OOPS-1113EB6 (bug 276144)
  31 <ubottu> Launchpad bug 320293 in launchpad-bazaar "OOPS traversing to branch on person page" [Medium,Confirmed] https://launchpad.net/bugs/320293
  32 <ubottu> Launchpad bug 276144 in launchpad-bazaar "OOPS when trying to delete stacked-on branches." [Medium,Triaged] https://launchpad.net/bugs/276144
  33 <Ursinha> 1 - I did that. From now on to discuss meeting time changes just add it to the list of meeting topics, on MeetingAgenda page
  34 <Ursinha> 2 - okay as well
  35 <Ursinha> next!
  36 <Ursinha> [TOPIC] * Oops report & Critical Bugs
  37 <MootBot> New Topic:  * Oops report & Critical Bugs
  38 <Ursinha> three oopses and one bug
  39 <Ursinha> the oopses
  40 <Ursinha> MissingMergeDirective: OOPS-1125CEMAIL1
  41 <Ursinha> this is yours rockstar?
  42 * Ursinha kicks ubottu 
  43 <rockstar> Ursinha, I can look at it
  44 <Ursinha> rockstar, thanks 
  45 <Ursinha> bac, to registry: OOPS-1125A165, OOPS-1125A1096 - this second one is the same in the email sinzui sent to barry
  46 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
  47 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
  48 * henninge (n=henning@i577A8E79.versanet.de) has joined #launchpad-meeting
  49 <henninge> me!
  50 <Ursinha> hi henninge!
  51 <henninge> Ursinha: hi! :)
  52 <bac> Ursinha: barry is working on the second one
  53 <henninge> sorry for the delay ...
  54 <Ursinha> bac, do you know if there's an open bug for that?
  55 * mars has quit (Remote closed the connection)
  56 <Ursinha> intellectronica, bug 279561, that's affecting people more frequently lately
  57 <ubottu> Launchpad bug 279561 in malone "No url for <Message at ...> when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561
  58 * intellectronica shoegazing
  59 <bac> Ursinha: i don't.  i'll check with barry when he returns from afk
  60 <intellectronica> sorry, it's still a mystery
  61 <intellectronica> i'll go back to working on it now
  62 <Ursinha> intellectronica, do you have time to deal with that? it's causing a lot of oopses and annoying people
  63 <Ursinha> bac, thanks 
  64 <intellectronica> Ursinha: i do now
  65 <Ursinha> [action] bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one
  66 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
  67 <MootBot> ACTION received:  bac to check with barry if there's an open bug for  OOPS-1125A1096, if not, Ursinha to file one
  68 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
  69 <Ursinha> bac, about the other one, do you have any clues?
  70 <Ursinha> intellectronica, thanks
  71 <Ursinha> [action] intellectronica to work on bug 279561
  72 <MootBot> ACTION received:  intellectronica to work on bug 279561
  73 <ubottu> Launchpad bug 279561 in malone "No url for <Message at ...> when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561
  74 <Ursinha> [action] rockstar to check OOPS-1125CEMAIL1
  75 <MootBot> ACTION received:  rockstar to check OOPS-1125CEMAIL1
  76 <Ursinha> any other issues you may want to point?
  77 <bac> Ursinha: no clue ATM.  will check it out
  78 <Ursinha> thanks bac
  79 <Ursinha> I appreciate that
  80 <Ursinha> [action] bac to take a look at OOPS-1125A165
  81 <MootBot> ACTION received:  bac to take a look at OOPS-1125A165
  82 <Ursinha> well, moving on
  83 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
  84 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
  85 <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
  86 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
  87 <herb> * 2008-01-28 - Yesterday - We rolled out 2.2.1. Total downtime was approximately 35 minutes.
  88 <herb> * Even after the rollout, we continue to have problems as a result of bug #156453 and bug #118625. I cannont emphasize enough how much of a pain point this is becoming for the LOSAs. More than a quarter of the incidents that we've had to deal with in the last year have been related to these two bugs.
  89 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
  90 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
  91 <herb> * We had a couple of incidents in the last week relating to bug #260171.
  92 <ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
  93 <herb> * Do we expect to roll out any additional patches this week? If so, when?
  94 <Ursinha> herb, I think at least one, I'll talk to kiko and confirm that to you
  95 <herb> ok
  96 <herb> Ursinha: do you suspect it will be rolled out today or tomorrow?
  97 <henninge> Ursinha: He approved one of mine, too.
  98 <Ursinha> [action] Ursinha to check with kiko if any other rollouts will happen this week
  99 <MootBot> ACTION received:  Ursinha to check with kiko if any other rollouts will happen this week
 100 <herb> I can wait, I was just curious if you have a best guess.
 101 <Ursinha> herb, guess tomorrow, but it depends on if we're having more issues to be fixed
 102 <herb> ok
 103 <herb> thanks
 104 <Ursinha> henninge, cool
 105 <Ursinha> thanks herb 
 106 <Ursinha> I took note of the "pain to the LOSAs" about that two bugs
 107 <herb> Ursinha: thanks. much appreciated
 108 <Ursinha> well, next
 109 <Ursinha> [TOPIC] * DBA report (DBA contact)
 110 <Ursinha> stub sent one email about it:
 111 <Ursinha> Looks like the upgrade went smoothly and nothing exciting is happening. Yay.
 112 <Ursinha> Although neither db server is particularly heavily loaded on average, we do see spikes sometimes. Also, the slave is still more loaded than production. As I don't expect us to get a second slave in the near future, I'm thinking we need to look into some better load balancing to move some of the slave load back to the master.
 113 <Ursinha> Config changes on one or two of the appserver instances to use the master as the slave would work in the short term. Long term I think we could be better with a load balancer like pgbouncer or pgpool (1 or 2).
 114 <MootBot> New Topic:  * DBA report (DBA contact)
 115 <Ursinha> that's all from stub
 116 <Ursinha> anything else?
 117 <Ursinha> anybody wants to point or discuss something?
 118 <Ursinha> 5
 119 <Ursinha> 4
 120 <Ursinha> 3
 121 <Ursinha> 2
 122 <Ursinha> 1
 123 <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
 124 <Ursinha> #endmeeting