1 <matsubara> #startmeeting
   2 <MootBot> Meeting started at 10:00. The chair is matsubara.
   3 <MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
   4 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
   5 <matsubara> [TOPIC] Roll Call 
   6 <MootBot> New Topic:  Roll Call
   7 <henninge> me
   8 <herb> me
   9 <bigjools> me
  10 <Ursinha> me
  11 <mars> me
  12 <abentley> me
  13 <sinzui> me
  14 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  15 <matsubara> hi stub 
  16 <stub> me
  17 <matsubara> let's move on, tom berger can join in later
  18 <matsubara> [TOPIC] Agenda 
  19 <MootBot> New Topic:  Agenda
  20 <matsubara>  * Actions from last meeting
  21 <matsubara>  * Oops report & Critical Bugs & Broken scripts
  22 <matsubara>  * Operations report (mthaddon/herb/spm)
  23 <matsubara>  * DBA report (stub)
  24 <matsubara> [TOPIC] * Actions from last meeting
  25 <MootBot> New Topic:  * Actions from last meeting
  26 <matsubara>  * Ursinha to check if all accounts requesting fix on feedback@ were fixed
  27 <matsubara>  * matsubara to chase rockstar about failure on updatebranches script
  28 <Ursinha> matsubara, apparently all accounts were fixed
  29 <matsubara> I talked to rockstar about the updatebranches script failures, he was debugging. I'll ask him again when he comes back from holidays
  30 <matsubara> [action] matsubara to chase rockstar about failure on updatebranches script
  31 <MootBot> ACTION received:  matsubara to chase rockstar about failure on updatebranches script
  32 <matsubara> thanks Ursinha 
  33 <matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
  34 <MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
  35 <matsubara> Ursinha, please take the stage
  36 <Ursinha> ok
  37 <Ursinha> most of the "unusual" oopses we had were rollout glitches, also we had a few very low priority which I talked to people and filed bugs
  38 <Ursinha> we have two critical bugs, after the rollout
  39 <Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403283
  40 <Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403306
  41 <ubottu> Error: This bug is private
  42 <ubottu> Ubuntu bug 403306 in launchpad-foundations "missing softlinks for shipit and login" [Critical,In progress]
  43 <matsubara> Ursinha, the private one is in progress but not assigned?
  44 <matsubara> who's taking care of it?
  45 <Ursinha> I wasn't sure if that was ok for the librarian bug to be public, given the info in the report, if so I'll be happy to switch
  46 <Ursinha> matsubara, apparently stub changed the status
  47 <Ursinha> stub, are you taking care of that?
  48 <matsubara> I assigned that one to stub 
  49 * bigjools-afk (n=quassel@82-71-93-254.dsl.in-addr.zen.co.uk) has joined #launchpad-meeting
  50 <Ursinha> about the missing softlinks, I know stub is taking care of that, but I have a question
  51 <stub> I have a branch awaiting RC
  52 <Ursinha> how could we avoid that to happen?
  53 * Received a CTCP PING 1248361621 from abentley
  54 <Ursinha> I see that the revision that caused that wasn't QAd - at least in the testplan that was NEEDSTESTING
  55 <stub> However, the production side of things needs to be done by the losas - editing the /etc/init.d script etc.
  56 <matsubara> [action] stub to get RC for branch that fixes bug 403283
  57 <MootBot> ACTION received:  stub to get RC for branch that fixes bug 403283
  58 <ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
  59 <ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
  60 <herb> stub: is the branch listed in the bug? I'd like to take a look at it so we know what to expect.
  61 * SteveA has quit (wolfe.freenode.net irc.freenode.net)
  62 * bigjools has quit (wolfe.freenode.net irc.freenode.net)
  63 <stub> https://code.edge.launchpad.net/~stub/launchpad/trivial/+merge/9179
  64 <stub> herb: it should be linked. You don't need that script of course, but it makes sense to keep it in the lp tree so devs can test if they broke it
  65 <herb> stub: cool. thanks. I'll take a look.
  66 <matsubara> Ursinha, are we expecting some of those low importance oops fixes to land for the re-roll?
  67 <mars> strange, the commit message doesn't mention having to touch those two links
  68 <Ursinha> matsubara, nope
  69 * SteveA (n=steve@canonical/stevea) has joined #launchpad-meeting
  70 <mars> "List
  71 <mars>         duplicate subscribers in the bug subscribers portlet even when
  72 <mars>         there is already an indirect subscription via a team membership."
  73 <Ursinha> matsubara, not so far
  74 <matsubara> Ursinha, isn't that what kiko asked yesterday?
  75 <stub> herb: And feel free to jump in and change and implement stuff if you want - lp-foundations doesn't have to own this stuff
  76 <Ursinha> matsubara, nope
  77 * bigjools-afk is now known as bigjools
  78 <stub> mars: I think it was a cock up rather than a deliberate change
  79 <Ursinha> but we have some outstanding oopses that keep happening
  80 <Ursinha> who's on behalf of flacoste today?
  81 <matsubara> Ursinha, yes, those are the ones I meant
  82 <matsubara> Ursinha, mars 
  83 <mars> stub, agreeded, but if so, did it just happen to slip past the dev, and the rc-reviewer?
  84 <stub> https://code.edge.launchpad.net/~stub/launchpad/pending-db-changes/+merge/9188 is also awaiting rc anyway. What I'm more interested in is how come the test suite still passes.
  85 <mars> Ursinha, I'm standing in for flacoste
  86 <matsubara> stub, why the branch that introduced the critical bug wasn't QA'd on staging?
  87 <Ursinha> matsubara, I guess that was a bugs branch
  88 <matsubara> oh, sorry. I thought it was a foundations issue
  89 <mars> stub, has the test suite changed to accommodate open sourcing in some way?  And thus removing those components from the suite?
  90 <Ursinha> matsubara, the missing links are, but which introduced that not
  91 <matsubara> Ursinha, ok.
  92 <stub> mars: I don't know.
  93 <Ursinha> mars, we have an oopses that happens almost everyday
  94 <Ursinha> *an oops
  95 <Ursinha> mars, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1300XMLP5
  96 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
  97 <Ursinha> mars, once I talked with flacoste about them, and he said that basically that wasn't our problem, iirc :)
  98 <mars> Ursinha, who would be the 'us' in "our problem"?
  99 <Ursinha> mars, foundations/launchpad
 100 <Ursinha> mars, i.e. not a bug
 101 <Ursinha> do you know what could we do to avoid those?
 102 <abentley> Ursinha: is this the private or public XMLRPC server?
 103 <mars> Ursinha, do we have a procedure for dealing with such noise?  The feels like a 404: it is an error in the system state, but not a programmer error.
 104 <Ursinha> I mean, we have a list of oopses that are growing every day because of hanging little things
 105 <Ursinha> mars, that's exactly my question :)
 106 <mars> Ursinha, for this specific issue, I would have to investigate.
 107 <Ursinha> mars, if we could deal differently those oopses, raising 404 instead of that oops, or even if we (me and matsubara) could move them to another section of the report
 108 <matsubara> abentley, private one
 109 <mars> for the larger issue of noise, well, maybe creating a new OOPS category or tag that can filter them out
 110 <stub> matsubara: The branch was rc, so not much QA time. Even if it was qa'd, that doesn't mean anyone would have tested the login or shipit systems which should have been unaffected according to the commit message.
 111 <abentley> matsubara: Well, it kinda is our problem, since we control both ends, then.
 112 <mars> Ursinha, not 404, but one of the other status codes, for "Ill-formed request"
 113 <Ursinha> mars, sorry, bad brain link :)
 114 <abentley> I agree that bad input should not be treated as an OOPS
 115 <matsubara> I don't like moving those oopses to another section.
 116 <Ursinha> matsubara, me neither.
 117 <abentley> But it should be logged *somewhere*.
 118 <matsubara> if the input is bad and we know it, let's not log an oops for it
 119 <Ursinha> mars, could you do some more investigation on this? I'll file a bug, can I?
 120 <matsubara> we used to have a bug for ExpatErrors
 121 * Ursinha looks
 122 <matsubara> was it fixed?
 123 <mars> Ursinha, certainly, please do
 124 <Ursinha> mars, ok, so I'll search for the bug, and file one if not able to find
 125 <matsubara> [action] ursinha do file bug for OOPS-1300XMLP5
 126 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
 127 <MootBot> ACTION received:  ursinha do file bug for OOPS-1300XMLP5
 128 <ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
 129 <Ursinha> thanks matsubara
 130 <matsubara> Ursinha, anything else?
 131 <mars> I think the procedure is to turn bad data from a "500: Internal Server Error" into a "400: Bad Request"
 132 <Ursinha> matsubara, yes
 133 <Ursinha> copy and paste fail
 134 <Ursinha> we also have constant UnicodeDecodeErrors and the like
 135 <Ursinha> everyday, and I noticed that they're somehow growing
 136 <Ursinha> is the unicode problem really unsolvable?
 137 <abentley> Ursinha: A bunch of them should disappear as of this rollout.
 138 <Ursinha> abentley, right, but they seem to be everywhere. I'll keep one eye on them, and will report back next meeting
 139 <Ursinha> matsubara, [action] Ursinha to ^
 140 <Ursinha> :P
 141 <Ursinha> mars, this one is hanging for some time: bug 354593
 142 <mars> Ursinha, a unicode error in Python is solvable.  Are they coming from everywhere in the system?
 143 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
 144 <abentley> Ursinha: If you're looking for an ultimate solution, it's along the lines of "use Python 3.0".
 145 <matsubara> [action] Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting
 146 <MootBot> ACTION received:  Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting
 147 <Ursinha> abentley, :)
 148 <Ursinha> mars, I'll show you later, we have several oopses and different bugs for them
 149 <Ursinha> 5 digit bugs :)
 150 <Ursinha> mars, can you take a look in that bug, please?
 151 <mars> ok
 152 <Ursinha> using kiko's words: "if they happen more than once a week they are not low priority"
 153 <Ursinha> matsubara, [action] mars to take a look at bug 354593
 154 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
 155 <mars> ok, so we know the problem, and we have an ideal solution
 156 <matsubara> [action] mars to take a look at bug 354593
 157 <MootBot> ACTION received:  mars to take a look at bug 354593
 158 <mars> did Francis mention a quick interim solution?
 159 <mars> stub, ^?
 160 <stub> interim solution of what?
 161 <Ursinha> stub, bug 354593
 162 <ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
 163 <stub> No - I haven't discussed that with him.
 164 <mars> ok
 165 <mars> I'll have a look at the reports, split them into new bugs, try to eliminate the OOPSes
 166 <Ursinha> mars, can you make comments in the bug, if that applies, please?
 167 <stub> It should be possible to register new exception views for that layer, no?
 168 <mars> the branding is a related, but larger issue
 169 <mars> Ursinha, sure
 170 <matsubara> Ursinha, anything else? we need to move on as we only have 11min
 171 <Ursinha> matsubara, no, I'm done for today.
 172 <matsubara> thanks everyone
 173 <Ursinha> thanks mars, stub, abentley and herb
 174 <matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
 175 <MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
 176 <herb> 2009-07-19 - We had another app server hang during log rotation. Fortunately it seems there has been some progress on bug #287304
 177 <Ursinha> and all the others :)
 178 <ubottu> Launchpad bug 287304 in launchpad-foundations "App Servers: Remove need for restart on logrotation" [High,Incomplete] https://launchpad.net/bugs/287304
 179 <herb> 2009-07-21 - We open sourced! Congratulations to all the developers on a job well done. We saw an uptick in traffic to the codehosting system and, apparently, quite a few new project registrations. Overall the system handled the increased load well.
 180 <herb> 2009-07-22 - Yesterday evening we rolled out 2.2.7. We had some hiccups again but we have bugs filed for those, as was discussed during the critical bugs section.
 181 <herb> That's it from the LOSAs unless there are questions.
 182 <matsubara> thanks herb 
 183 <matsubara> [TOPIC] * DBA report (stub)
 184 <MootBot> New Topic:  * DBA report (stub)
 185 <herb> thanks matsubara
 186 <stub> Database update went painlessly, or at least I haven't heard any tales of pain and misery and nobody woke me up.
 187 <stub> With 2.2.7, we can start pruning unwanted People. We just need to add '--experimental' to garbo-daily.py and maybe tweak the --abort-script argument if the default of 24 hours is too long.
 188 <stub> Nothing else interesting happening.
 189 <matsubara> stub, are you taking care of enabling the pruning script? or have an RT for the losas to do it?
 190 <mthaddon> stub: yep, db updates were pretty quick and painless this time around
 191 <stub> I think salgado might have done that already. I was going to worry about it after reroll and the fires are all out.
 192 <matsubara> [action] matsubara to chase salgado about people pruning script
 193 <MootBot> ACTION received:  matsubara to chase salgado about people pruning script
 194 <matsubara> ok, anything else for stub?
 195 <matsubara> thanks stub 
 196 <matsubara> before I close, just want to let you know that I created the 2.2.8 milestone
 197 <matsubara> during yesterday's TLs meeting, the TL said it would be useful to coordinate work even if we're not doing a release for 2.2.8
 198 <matsubara> so please, since we're not doing the 2.2.8 release, don't let your QA itens otherwise will have a huge backlog for 2.2.9
 199 <matsubara> s/itens/items slip/
 200 <matsubara> I think that's all for today
 201 <matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 202 <matsubara> #endmeeting 
 203 <MootBot> Meeting finished at 10:43.