1 <Ursinha> #startmeeting
2 <Ursinha> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
3 <Ursinha> [TOPIC] Roll Call
4 <Ursinha> Not on the Launchpad Dev team? Welcome! Come "me" with the rest of us!
5 <MootBot> Meeting started at 10:00. The chair is Ursinha.
6 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
7 <MootBot> New Topic: Roll Call
8 <bigjools> me
9 <Ursinha> me :)
10 <flacoste> me
11 <sinzui> me
12 <henninge> me
13 <rockstar> me
14 <Ursinha> herb: hi
15 <rockstar> I wish someone not on the dev team would "me"
16 <Ursinha> rockstar: me too
17 <Ursinha> anyway :)
18 <intellectronica> rockstar, Ursinha: you can get that, but it requires /a lot/ of herding
19 <Ursinha> herb: are you there by chance? :)
20 <Ursinha> intellectronica: indeed
21 -!- stub
22 <Ursinha> so here is stub
23 <Ursinha> only herb missing
24 <herb> me
25 <herb> sorry
26 <Ursinha> :)
27 <Ursinha> hi herb!
28 <Ursinha> moving on :)
29 <herb> hi
30 <Ursinha> [TOPIC] Agenda * Actions from last meeting * Oops report & Critical Bugs * Operations report (mthaddon/herb/spm) * DBA report (stub)
31 <MootBot> New Topic: Agenda * Actions from last meeting * Oops report & Critical Bugs * Operations report (mthaddon/herb/spm) * DBA report (stub)
32 <Ursinha> argh
33 <Ursinha> [TOPIC] * Actions from last meeting
34 <Ursinha> • sinzui to request CP for bugs 341085 and 341117
35 <Ursinha> • ursinha to talk to allenap/bjornt about a bug marked as won't fix, but had another OOPS occurrence: bug 341766 - done
36 <Ursinha> • ursinha to debug 340058 and write the results back to the report - done
37 <Ursinha> • bigjools to chase kiko or flacoste to approve CPs. matsubara to remind him about it :-)
38 <MootBot> New Topic: * Actions from last meeting
39 <ubottu> Bug 341085 on http://launchpad.net/bugs/341085 is private
40 <ubottu> Launchpad bug 341766 in malone "AttributeError OOPS when receiving data from apport on +filebug" [Low,Triaged] https://launchpad.net/bugs/341766
41 <Ursinha> my items are done
42 <bigjools> done
43 <sinzui> Ursinha: Done
44 <Ursinha> awesome!
45 <Ursinha> next
46 <Ursinha> [TOPIC] * Oops report & Critical Bugs
47 <Ursinha> intellectronica, https://bugs.edge.launchpad.net/malone/+bug/329908
48 <Ursinha> rockstar, as we previously talked, https://bugs.edge.launchpad.net/launchpad-bazaar/+bug/328302
49 <MootBot> New Topic: * Oops report & Critical Bugs
50 <ubottu> Ubuntu bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New]
51 <ubottu> Error: This bug is private
52 <Ursinha> is it?
53 * Ursinha looks again
54 * rockstar doesn't think that's the right bug...
55 <intellectronica> Ursinha: no news about this one yet :(
56 <rockstar> Ursinha, yeah, it's private because it has query info
57 <Ursinha> bug 328302
58 <ubottu> Bug 328302 on http://launchpad.net/bugs/328302 is private
59 <Ursinha> it's this one
60 <Ursinha> rockstar: I told you about that a few mins ago :)
61 <rockstar> Ursinha, yeah, and it looks like it's In Progress.
62 <Ursinha> intellectronica: that bug is happening like everyday
63 <Ursinha> rockstar: I brought it here because you didn't comment on that
64 <Ursinha> rockstar: at -code, that is
65 <rockstar> Ursinha, I have nothing to comment on. Looks like thumper is working on it.
66 <Ursinha> rockstar: so as I presumed, you know as much about that as I do
67 <rockstar> Ursinha, unfortunately, due to illness and travelling, I've been a bit out of touch.
68 <intellectronica> Ursinha: shit. ok, i'll try to tackle it soon
69 <Ursinha> rockstar: no problem :) just brought to attention to see if you had some info that I (neither the bug report) would have
70 <Ursinha> intellectronica: thanks for that
71 <Ursinha> [action] intellectronica to make efforts to take a look at bug 329908
72 <MootBot> ACTION received: intellectronica to make efforts to take a look at bug 329908
73 <ubottu> Launchpad bug 329908 in malone "DownloadFailed OOPS when reporting a bug with apport" [Undecided,New] https://launchpad.net/bugs/329908
74 <Ursinha> :)
75 <intellectronica> :)
76 <Ursinha> about the code bug, I'll catch up with thumper later, if the status doesn't change
77 <Ursinha> thanks intellectronica and rockstar
78 <rockstar> Ursinha, he commented and said he had a fix for it.
79 <rockstar> Takes the query down to 3s instead of 17s.
80 <Ursinha> rockstar: I saw that.
81 <Ursinha> thanks
82 <Ursinha> moving on
83 <Ursinha> all critical bugs are fix committed, yay!
84 <Ursinha> that's all for me in oops/bugs section
85 <Ursinha> next
86 <Ursinha> [TOPIC] * Operations report (mthaddon/herb/spm)
87 <MootBot> New Topic: * Operations report (mthaddon/herb/spm)
88 <herb> 2009-03-20 - CP r7866 and r7900 to production.
89 <herb> Sounding like a broken record, bug 156453 and bug 118625 continue to be a problem.
90 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,Triaged] https://launchpad.net/bugs/156453
91 <herb> There have been some problems with staging updates this week. Seem to be mostly DB related and resolved now. There is a staging update running now.
92 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
93 <herb> sinzui has a couple of cherry pick requests pending approval.
94 <herb> Other than that it's been a quiet week. Unless there are any questions, that's it from the LOSAs unless there are questions.
95 <rockstar> herb, we may actually have an answer for you next week! :)
96 * sinzui will speak with kiko.
97 <herb> rockstar: that is exciting news
98 <Ursinha> sinzui: those are the ones of the actions from last meeting?
99 <sinzui> I'm not certain those items are worthy of a CP
100 <rockstar> herb, mwhudson and I are going to be hacking together on this soon.
101 <Ursinha> [action] sinzui to talk to kiko about pending cp requests
102 <MootBot> ACTION received: sinzui to talk to kiko about pending cp requests
103 <sinzui> hence those bugs were high
104 -!- SteveA
105 <Ursinha> sinzui: hm, I see
106 <Ursinha> sinzui: are you going to sort out with kiko?
107 <sinzui> I'll get his opinion
108 <Ursinha> rockstar: that's cool
109 <Ursinha> right sinzui, thanks
110 <Ursinha> anything else for herb?
111 <Ursinha> that sounds a no :)
112 <Ursinha> thanks herb
113 <herb> thanks Ursinha
114 <Ursinha> moving on
115 <Ursinha> [TOPIC] * DBA report (stub)
116 <MootBot> New Topic: * DBA report (stub)
117 <stub> If you see 'database in recovery mode', it means part of PostgreSQL has crashed. We have seen it this week on staging and twice with buildbot. We don't know the trigger yet. PostgreSQL will recover, but our code might not. When it happened on staging, it looks like the staging appserver didn't reconnect fully and needed to be manually restarted.
118 <stub> If anyone has statement logging on and triggers the bug, I'l like your log files.
119 <stub> I'll abuse an ec2 instance tomorrow in an attempt to trigger the bug and try to get a pointer to the trigger.
120 <stub> A new PostgreSQL update came through today which contains some crash fixes. With luck, we will upgrade and the problem will never be seen again.
121 <stub> All the bugs I can find in the updated database maintenance tools have been fixed, so staging updates should work fine now (barring the PostgreSQL crash).
122 <stub> Some bigish DB changes are part of the next rollout. We are now maintaining two slaves so the maintenance tools needed to be updated to cope. We have also split our single replication set into two now as part of keeping the SSO service running during Launchpad outages. With luck, nobody will notice and things will continue to run smoothly.
123 <stub> I'm done
124 <Ursinha> hm.
125 <Ursinha> anyone has questions for stub?
126 <Ursinha> stub: that happened only on staging, right?
127 <stub> Crash on staging and twice on ec2 with buildbot
128 <Ursinha> this explain "some" weird oopses
129 <Ursinha> hm, thanks stub
130 <Ursinha> anyone else wants to say something?
131 <Ursinha> 3
132 <Ursinha> 2
133 <Ursinha> 1
134 <Ursinha> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs.
135 <Ursinha> #endmeeting
136 <MootBot> Meeting finished at 10:18.
launchpad development