1 <matsubara> #startmeeting
2 <MootBot> Meeting started at 10:00. The chair is matsubara.
3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
5 <matsubara> [TOPIC] Roll Call
6 <MootBot> New Topic: Roll Call
7 <gary_poster> me
8 <rockstar> ni! ni!
9 <danilos> me
10 <allenap> me
11 <sinzui> me
12 <matsubara> Chex, bigjools, hi
13 <bigjools> me
14 <Chex> Chex: hello
15 <matsubara> ok, everyone is here.
16 <Chex> .. /o\ err, hi
17 <matsubara> apologies from Ursinha and stub
18 <matsubara> [TOPIC] Agenda
19 <MootBot> New Topic: Agenda
20 <matsubara> * Actions from last meeting
21 <matsubara> * Oops report & Critical Bugs & Broken scripts
22 <matsubara> * Operations report (mthaddon/Chex/spm/mbarnett)
23 <matsubara> * DBA report (stub)
24 <matsubara> * Proposed items
25 <matsubara> [TOPIC] * Actions from last meeting
26 <MootBot> New Topic: * Actions from last meeting
27 <matsubara> * Ursinha to send one email to lp list explaining the qa-tags experiment
28 <matsubara> * Chex to follow up with thumper about the multiple git import failures on the importd
29 <matsubara> * matsubara to file a high/critical bug for OOPS-1430F2574
30 <matsubara> * matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1427EA45
31 <matsubara> * matsubara to email tim about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1426EC1536
32 <matsubara> * emailed Tim about it
33 <matsubara> * matsubara to talk to TL about not having the LP production meeting anymore or change its format
34 <matsubara> * matsubara to email QA contacts about next LP prod. meeting at 16UTC
35 <matsubara> * emailed the list and QA contacts about this :-)
36 <matsubara> * matsubara to email losas about their weekly report
37 <matsubara> * emailed them requesting that the operation report be sent to the list
38 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
39 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1427EA45
40 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1426EC1536
41 <matsubara> I talked to salgado about OOPS-1430F2574 and it wasn't necessary to file a bug for that one. it was a one off problem and I'm keeping an eye on oops reports if it shows up again
42 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1430F2574
43 <matsubara> danilos sent the email about the qa-tags experiment
44 <danilos> matsubara, Ursula is on vacation, I'd like her to give us her PoV as well
45 <matsubara> Chex, did you sort out the failures in the git import with thumper?
46 <danilos> matsubara, though, that will likely happen as part of the discussion on the list, so we can probably take the action item off
47 <matsubara> danilos, right. thanks for starting the discussion
48 <Chex> matsubara: I followed up with him briefly, but was not able to resolve anything, I need to talk to him again, sorry about that
49 <danilos> np, it was way overdue
50 <matsubara> Chex, shall I re-add the action item to the list?
51 <Chex> matsubara: yes please do
52 <matsubara> [action] * Chex to follow up with thumper about the multiple git import failures on the importd
53 <MootBot> ACTION received: * Chex to follow up with thumper about the multiple git import failures on the importd
54 <matsubara> ok, thanks Chex
55 <matsubara> let's move on
56 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
57 <MootBot> New Topic: * Oops report & Critical Bugs & Broken scripts
58 <matsubara> bigjools, https://bugs.edge.launchpad.net/soyuz/+bug/493703
59 <ubottu> Ubuntu bug 493703 in soyuz "LocationError raised in build page and distribution arch series binary package page" [High,Triaged]
60 <matsubara> it's targeted to .12, currently not assigned and the cycle will end tomorrow. Any chance of have that one fixed before the holidays? it's generating > 1K OOPS a day (most of them from robots but it's pretty importante nonetheless)
61 <bigjools> matsubara: zero chance
62 <matsubara> s/importante/important/ sorry for the portuguese leakage there :-)
63 <bigjools> heh I am used to it from working with cprov :)
64 <matsubara> bigjools, :-(
65 <matsubara> the sad smile is for the zero chance comment btw
66 <bigjools> yeah, there's another serious problem that is taking precedence. If by some miracle I get that fixed then we can look at the oopses
67 <matsubara> hmm that's the top OOPS we have
68 <bigjools> gar sorry
69 <bigjools> when will the pain end this week
70 <matsubara> oh, the retry dep thingie?
71 <bigjools> yep
72 <matsubara> right. ok then
73 <matsubara> gary_poster, https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403618
74 <ubottu> Ubuntu bug 403618 in launchpad-foundations "Launchpad should return a 404 instead of ForbiddenAttribute OOPS" [High,Triaged]
75 <matsubara> gary_poster, same thing, that one is happening quite frequently. any chance of landing a fix before the holidays?
76 <gary_poster> matsubara: holidays, yes, next release, no
77 <gary_poster> where yes is "any chance" :-)
78 <gary_poster> I suppose it can be an RC then
79 <gary_poster> I mean CP
80 <matsubara> gary_poster, all right. as long as they disappear from the OOPS summaries, it's good :-)
81 <gary_poster> :-) understood
82 <matsubara> gary_poster, could you take a look at https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 ?
83 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
84 <matsubara> it's a timeout error on the api
85 <matsubara> I'm not sure if it's just regular timeouts, if they do, then I'd need to update oops-tools to handle those just like any other timeouts
86 <matsubara> currently they show up in the exceptions section
87 <gary_poster> matsubara: yes, it's another timeout
88 <matsubara> rockstar, OOPS-1438EA844
89 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1438EA844
90 <matsubara> gary_poster, so just a matter of moving that kind of exception to the right section in the summaries?
91 <gary_poster> matsubara: i.e., this is something that should be addressed by bugs, notfoundatons/leonardr
92 * rockstar looks
93 <matsubara> gary_poster, ok, looks like a time out using the dupe finder
94 <matsubara> but using the API
95 <matsubara> so, I'll talk to the bugs team about it and sort it out (and file a bug to have oops-tools updated to handle it appropriately)
96 <gary_poster> matsubara: right. the problem probably needs to be addressed in lp.bugs.model.bugtask, line 571, in findSimilarBugs
97 <matsubara> sinzui, https://bugs.edge.launchpad.net/launchpad-registry/+bug/495051
98 <ubottu> Ubuntu bug 495051 in launchpad-registry "UnboundLocalError editing proposed team membership" [High,In progress]
99 <sinzui> 'nough said
100 <matsubara> allenap, I have a few timeout OOPSes on +filebug. are you interested in those? I know gmb just landed code to make it async so maybe it's just a matter of waiting for that and see how things will behave
101 <matsubara> sinzui, indeed! thanks dude!
102 <danilos> matsubara, how common is eg. bug 493703 in OOPSes? it looks reasonably simple to solve that somebody outside soyuz can fix it? bigjools, is my assessment wrong?
103 <ubottu> Launchpad bug 493703 in soyuz "LocationError raised in build page and distribution arch series binary package page" [High,Triaged] https://launchpad.net/bugs/493703
104 <allenap> matsubara: gmb's async stuff probably won't make the timeouts any different, it's just that the user won't be so affected.
105 <bigjools> danilos: noodles is going to look into it
106 <matsubara> [action] matsubara to talk to bugs team about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 and file a bug on oops-tools to handle LaunchpadTimeoutError correctly
107 <MootBot> ACTION received: matsubara to talk to bugs team about https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1439EB784 and file a bug on oops-tools to handle LaunchpadTimeoutError correctly
108 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
109 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1439EB784
110 <allenap> matsubara: The URL will change slightly, to +filebug-inline-form. Timeouts for this page are far less important.
111 <danilos> bigjools, ok, if you need help finding someone to work on it (though, looking into it might be most of the work anyway), I'd be happy to give a hand tomorrow (in looking for someone, not doing it :)
112 <bigjools> danilos: ok thanks :)
113 <matsubara> danilos, thanks. we have around 1K of those a day (mostly from bots triggering it)
114 <danilos> matsubara, right, thanks
115 <matsubara> allenap, all right. yesterday bigkev was trying to file some bugs and couldn't due to timeouts. I wonder if you need more OOPSes to help investigate the issue
116 <allenap> matsubara: He should be able to file bugs today, because the async dupe-finder is there now. But more OOPS reports are useful, if you have a bug to attach them to?
117 <matsubara> allenap, cool. I'll add those to the bug report then
118 <allenap> matsubara: Thanks :)
119 <matsubara> [action] matsubara to add +filebug timeout oopses to the bug report
120 <MootBot> ACTION received: matsubara to add +filebug timeout oopses to the bug report
121 <matsubara> rockstar, that SQLObjectNotFound oops is quite strange
122 <matsubara> rockstar, have you seen it before? looks like it happened only twice
123 <rockstar> matsubara, yeah, I'm looking at it. It probably should have 404'd - Not sure though.
124 <matsubara> and I was unable to reproduce
125 <rockstar> I have not seen it before. It's probably some corrupted bmp somewhere.
126 <danilos> matsubara, that strikes me as replication-related, but I am no expert :)
127 <matsubara> rockstar, worth a bug rpeort for that one?
128 <rockstar> danilos, yeah, that's what I thought.
129 <rockstar> matsubara, not sure. I'll look into it, and file one if need be.
130 <matsubara> danilos, rockstar yeah, same thought here
131 <matsubara> rockstar, cool. thanks! I'll let you know if it happens again
132 <matsubara> we had some script failures since last week
133 <matsubara> Scripts failed to run: loganberry:allocate-revision-karma, loganberry:flag-expired-memberships
134 <matsubara> sinzui, ^ I think that one is yours?
135 <matsubara> the retry depwait script failure is being worked on by bigjools
136 <sinzui> matsubara: We have had intermittent timing issues because of long processes
137 <bigjools> floundering on
138 <sinzui> matsubara: there are no errors and the script do run when they get their turn
139 <matsubara> sinzui, ok, so that means that the failures on mizuho:librarian-gc and loganberry:karma-update, loganberry:allocate-revision-karma, loganberry:launchpad-stats, loganberry:expire-questions, loganberry:productreleasefinder, loganberry:update-cache, loganberry:launchpad-targetnamecacheupdate are probably related to that?
140 <matsubara> I guess so, since the last failures for those were 2 days ago
141 <matsubara> we have only one critical bug which bigjools is on it.
142 <sinzui> matsubara: right. I do not investigate a failure to run for 24 hours after the notice because ANOTHER process is responsible for that. When all scripts fail, I might investigate withing 24 hours
143 <matsubara> I see. all right then. thanks sinzui
144 <matsubara> and thanks everyone. let's move on
145 <matsubara> [TOPIC] * Operations report (mthaddon/Chex/spm/mbarnett)
146 <MootBot> New Topic: * Operations report (mthaddon/Chex/spm/mbarnett)
147 <sinzui> matsubara: and i check if we changed production code in the last 24 hours
148 <matsubara> Chex, ?
149 <Chex> hello everyone, a rport focused on the LP rollout:
150 <matsubara> wow, nice timing :-)
151 <Chex> - The LP 3.1.11 rollout was last week, and there is a upcoming 'short' LP rollout next week.
152 <Chex> - 3.1.11 roll-out took 2 days, due to some problems with the rollout
153 <Chex> process. We are working to address these issues for next time.
154 <Chex> Steps we are taking to improve the process are:
155 <Chex> : moving to build centrally before pushing code out to speed up pushing and building of code
156 <Chex> : investigating less error prone ways (and quicker ways) of switching to read-only mode
157 <Chex> : ensuring we're not interrupted by other DB jobs on other servers in the cluster that block the DB upgrade
158 <Chex> and thats all for this week, questions/comments, anyone??
159 <matsubara> Chex, thanks
160 * noodles775 has quit (Read error: 110 (Connection timed out))
161 <Chex> matsubara: your welcome
162 <matsubara> [TOPIC] * DBA report (stub)
163 <MootBot> New Topic: * DBA report (stub)
164 <danilos> Chex, yes, are we getting any of this for 3.1.12?
165 <matsubara> oops, sorry, go ahead danilos
166 <matsubara> [action] matsubara to email stub about the DBA report
167 * noodles775 (n=miken@canonical/launchpad/noodles775) has joined #launchpad-meeting
168 <MootBot> ACTION received: matsubara to email stub about the DBA report
169 <Chex> danilos: yes, most of those items I listed should make it into the 3.1.12 release, I believe
170 <danilos> Chex, ok, cool, that sounds great then, but there's always potential for failure with new features like that; I'll try to keep an eye on that :)
171 <Chex> danilos: ok, great, we appreciate all and any eyeballs on the process
172 <danilos> Chex, fwiw, I'll be doing a release manager rotation, it's not that I don't trust our lovely LOSA team :)
173 <danilos> bigjools, (add a link to the image :)
174 <bigjools> http://people.canonical.com/~ed/losa-team.png
175 <MootBot> LINK received: http://people.canonical.com/~ed/losa-team.png
176 <matsubara> LOL
177 <danilos> thank you, we can move on :)
178 <matsubara> sorry, got very distracted by that picture hehe
179 <matsubara> [TOPIC] * Proposed items
180 <MootBot> New Topic: * Proposed items
181 <matsubara> there's no new proposed items
182 <matsubara> the new meeting time seems to work fine for everyone
183 <matsubara> anything else before I close?
184 <danilos> And if anyone has any issues that may need tracking, please ping me as the release manager for 3.1.12. Thank you.
185 <bigjools> danilos, my hero
186 * salgado-lunch is now known as salgado
187 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See https://dev.launchpad.net/MeetingAgenda for the logs.
188 <matsubara> #endmeeting
189 <MootBot> Meeting finished at 10:37.