1 <Ursinha> #startmeeting
2 <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
3 <Ursinha> [TOPIC] Roll Call
4 <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
5 <MootBot> Meeting started at 09:02. The chair is Ursinha.
6 <MootBot> New Topic: Roll Call
7 <Ursinha> hi MootBot
8 <Ursinha> me
9 <herb> me
10 <bac> me
11 <intellectronica> me
12 <gary_poster> me
13 <Ursinha> rockstar, ping
14 <cprov> me
15 <Ursinha> henninge is mia
16 <rockstar> me (barely)
17 <Ursinha> :)
18 <Ursinha> well, no issues for translations, so no much of a problem
19 <Ursinha> alright
20 <Ursinha> [TOPIC] Agenda
21 <Ursinha> * Actions from last meeting
22 <Ursinha> * Oops report & Critical Bugs-
23 <Ursinha> * Operations report (mthaddon/herb/spm)
24 <Ursinha> * DBA report (DBA contact)
25 <MootBot> New Topic: Agenda
26 <Ursinha> [TOPIC] * Actions from last meeting
27 <MootBot> New Topic: * Actions from last meeting
28 <Ursinha> 1) matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time
29 <Ursinha> 2) TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week (26th to 30th Jan)
30 <Ursinha> 3) matsubara to file bugs for OOPS-1116EC265 (bug 320293), OOPS-1113EB6 (bug 276144)
31 <ubottu> Launchpad bug 320293 in launchpad-bazaar "OOPS traversing to branch on person page" [Medium,Confirmed] https://launchpad.net/bugs/320293
32 <ubottu> Launchpad bug 276144 in launchpad-bazaar "OOPS when trying to delete stacked-on branches." [Medium,Triaged] https://launchpad.net/bugs/276144
33 <Ursinha> 1 - I did that. From now on to discuss meeting time changes just add it to the list of meeting topics, on MeetingAgenda page
34 <Ursinha> 2 - okay as well
35 <Ursinha> next!
36 <Ursinha> [TOPIC] * Oops report & Critical Bugs
37 <MootBot> New Topic: * Oops report & Critical Bugs
38 <Ursinha> three oopses and one bug
39 <Ursinha> the oopses
40 <Ursinha> MissingMergeDirective: OOPS-1125CEMAIL1
41 <Ursinha> this is yours rockstar?
42 * Ursinha kicks ubottu
43 <rockstar> Ursinha, I can look at it
44 <Ursinha> rockstar, thanks
45 <Ursinha> bac, to registry: OOPS-1125A165, OOPS-1125A1096 - this second one is the same in the email sinzui sent to barry
46 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
47 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
48 * henninge (n=henning@i577A8E79.versanet.de) has joined #launchpad-meeting
49 <henninge> me!
50 <Ursinha> hi henninge!
51 <henninge> Ursinha: hi! :)
52 <bac> Ursinha: barry is working on the second one
53 <henninge> sorry for the delay ...
54 <Ursinha> bac, do you know if there's an open bug for that?
55 * mars has quit (Remote closed the connection)
56 <Ursinha> intellectronica, bug 279561, that's affecting people more frequently lately
57 <ubottu> Launchpad bug 279561 in malone "No url for <Message at ...> when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561
58 * intellectronica shoegazing
59 <bac> Ursinha: i don't. i'll check with barry when he returns from afk
60 <intellectronica> sorry, it's still a mystery
61 <intellectronica> i'll go back to working on it now
62 <Ursinha> intellectronica, do you have time to deal with that? it's causing a lot of oopses and annoying people
63 <Ursinha> bac, thanks
64 <intellectronica> Ursinha: i do now
65 <Ursinha> [action] bac to check with barry if there's an open bug for OOPS-1125A1096, if not, Ursinha to file one
66 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
67 <MootBot> ACTION received: bac to check with barry if there's an open bug for OOPS-1125A1096, if not, Ursinha to file one
68 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A1096
69 <Ursinha> bac, about the other one, do you have any clues?
70 <Ursinha> intellectronica, thanks
71 <Ursinha> [action] intellectronica to work on bug 279561
72 <MootBot> ACTION received: intellectronica to work on bug 279561
73 <ubottu> Launchpad bug 279561 in malone "No url for <Message at ...> when trying to access bug messages through the API" [Medium,Triaged] https://launchpad.net/bugs/279561
74 <Ursinha> [action] rockstar to check OOPS-1125CEMAIL1
75 <MootBot> ACTION received: rockstar to check OOPS-1125CEMAIL1
76 <Ursinha> any other issues you may want to point?
77 <bac> Ursinha: no clue ATM. will check it out
78 <Ursinha> thanks bac
79 <Ursinha> I appreciate that
80 <Ursinha> [action] bac to take a look at OOPS-1125A165
81 <MootBot> ACTION received: bac to take a look at OOPS-1125A165
82 <Ursinha> well, moving on
83 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
84 <ubottu> https://devpad.canonical.com/~jamesh/oops.cgi/1125A165
85 <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
86 <MootBot> New Topic: * Operations report (mthaddon/herb/spm)
87 <herb> * 2008-01-28 - Yesterday - We rolled out 2.2.1. Total downtime was approximately 35 minutes.
88 <herb> * Even after the rollout, we continue to have problems as a result of bug #156453 and bug #118625. I cannont emphasize enough how much of a pain point this is becoming for the LOSAs. More than a quarter of the incidents that we've had to deal with in the last year have been related to these two bugs.
89 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
90 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
91 <herb> * We had a couple of incidents in the last week relating to bug #260171.
92 <ubottu> Bug 260171 on http://launchpad.net/bugs/260171 is private
93 <herb> * Do we expect to roll out any additional patches this week? If so, when?
94 <Ursinha> herb, I think at least one, I'll talk to kiko and confirm that to you
95 <herb> ok
96 <herb> Ursinha: do you suspect it will be rolled out today or tomorrow?
97 <henninge> Ursinha: He approved one of mine, too.
98 <Ursinha> [action] Ursinha to check with kiko if any other rollouts will happen this week
99 <MootBot> ACTION received: Ursinha to check with kiko if any other rollouts will happen this week
100 <herb> I can wait, I was just curious if you have a best guess.
101 <Ursinha> herb, guess tomorrow, but it depends on if we're having more issues to be fixed
102 <herb> ok
103 <herb> thanks
104 <Ursinha> henninge, cool
105 <Ursinha> thanks herb
106 <Ursinha> I took note of the "pain to the LOSAs" about that two bugs
107 <herb> Ursinha: thanks. much appreciated
108 <Ursinha> well, next
109 <Ursinha> [TOPIC] * DBA report (DBA contact)
110 <Ursinha> stub sent one email about it:
111 <Ursinha> Looks like the upgrade went smoothly and nothing exciting is happening. Yay.
112 <Ursinha> Although neither db server is particularly heavily loaded on average, we do see spikes sometimes. Also, the slave is still more loaded than production. As I don't expect us to get a second slave in the near future, I'm thinking we need to look into some better load balancing to move some of the slave load back to the master.
113 <Ursinha> Config changes on one or two of the appserver instances to use the master as the slave would work in the short term. Long term I think we could be better with a load balancer like pgbouncer or pgpool (1 or 2).
114 <MootBot> New Topic: * DBA report (DBA contact)
115 <Ursinha> that's all from stub
116 <Ursinha> anything else?
117 <Ursinha> anybody wants to point or discuss something?
118 <Ursinha> 5
119 <Ursinha> 4
120 <Ursinha> 3
121 <Ursinha> 2
122 <Ursinha> 1
123 <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
124 <Ursinha> #endmeeting