## Template for LP Production Meeting logs. Just paste xchat log below and the format IRC line will take care of formatting correctly
#format IRC
<matsubara> #startmeeting
<MootBot> Meeting started at 10:00. The chair is matsubara.
<MootBot> Commands Available: [TOPIC], [IDEA], [ACTION], [AGREED], [LINK], [VOTE]
<matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues. 
<matsubara> [TOPIC] Roll Call 
<MootBot> New Topic:  Roll Call
<henninge> me
<herb> me
<bigjools> me
<Ursinha> me
<mars> me
<abentley> me
<sinzui> me
* stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
<matsubara> hi stub 
<stub> me
<matsubara> let's move on, tom berger can join in later
<matsubara> [TOPIC] Agenda 
<MootBot> New Topic:  Agenda
<matsubara>  * Actions from last meeting
<matsubara>  * Oops report & Critical Bugs & Broken scripts
<matsubara>  * Operations report (mthaddon/herb/spm)
<matsubara>  * DBA report (stub)
<matsubara> [TOPIC] * Actions from last meeting
<MootBot> New Topic:  * Actions from last meeting
<matsubara>  * Ursinha to check if all accounts requesting fix on feedback@ were fixed
<matsubara>  * matsubara to chase rockstar about failure on updatebranches script
<Ursinha> matsubara, apparently all accounts were fixed
<matsubara> I talked to rockstar about the updatebranches script failures, he was debugging. I'll ask him again when he comes back from holidays
<matsubara> [action] matsubara to chase rockstar about failure on updatebranches script
<MootBot> ACTION received:  matsubara to chase rockstar about failure on updatebranches script
<matsubara> thanks Ursinha 
<matsubara> [TOPIC] * Oops report & Critical Bugs & Broken scripts
<MootBot> New Topic:  * Oops report & Critical Bugs & Broken scripts
<matsubara> Ursinha, please take the stage
<Ursinha> ok
<Ursinha> most of the "unusual" oopses we had were rollout glitches, also we had a few very low priority which I talked to people and filed bugs
<Ursinha> we have two critical bugs, after the rollout
<Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403283
<Ursinha> https://bugs.edge.launchpad.net/launchpad-foundations/+bug/403306
<ubottu> Error: This bug is private
<ubottu> Ubuntu bug 403306 in launchpad-foundations "missing softlinks for shipit and login" [Critical,In progress]
<matsubara> Ursinha, the private one is in progress but not assigned?
<matsubara> who's taking care of it?
<Ursinha> I wasn't sure if that was ok for the librarian bug to be public, given the info in the report, if so I'll be happy to switch
<Ursinha> matsubara, apparently stub changed the status
<Ursinha> stub, are you taking care of that?
<matsubara> I assigned that one to stub 
* bigjools-afk (n=quassel@82-71-93-254.dsl.in-addr.zen.co.uk) has joined #launchpad-meeting
<Ursinha> about the missing softlinks, I know stub is taking care of that, but I have a question
<stub> I have a branch awaiting RC
<Ursinha> how could we avoid that to happen?
* Received a CTCP PING 1248361621 from abentley
<Ursinha> I see that the revision that caused that wasn't QAd - at least in the testplan that was NEEDSTESTING
<stub> However, the production side of things needs to be done by the losas - editing the /etc/init.d script etc.
<matsubara> [action] stub to get RC for branch that fixes bug 403283
<MootBot> ACTION received:  stub to get RC for branch that fixes bug 403283
<ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
<ubottu> Bug 403283 on http://launchpad.net/bugs/403283 is private
<herb> stub: is the branch listed in the bug? I'd like to take a look at it so we know what to expect.
* SteveA has quit (wolfe.freenode.net irc.freenode.net)
* bigjools has quit (wolfe.freenode.net irc.freenode.net)
<stub> https://code.edge.launchpad.net/~stub/launchpad/trivial/+merge/9179
<stub> herb: it should be linked. You don't need that script of course, but it makes sense to keep it in the lp tree so devs can test if they broke it
<herb> stub: cool. thanks. I'll take a look.
<matsubara> Ursinha, are we expecting some of those low importance oops fixes to land for the re-roll?
<mars> strange, the commit message doesn't mention having to touch those two links
<Ursinha> matsubara, nope
* SteveA (n=steve@canonical/stevea) has joined #launchpad-meeting
<mars> "List
<mars>         duplicate subscribers in the bug subscribers portlet even when
<mars>         there is already an indirect subscription via a team membership."
<Ursinha> matsubara, not so far
<matsubara> Ursinha, isn't that what kiko asked yesterday?
<stub> herb: And feel free to jump in and change and implement stuff if you want - lp-foundations doesn't have to own this stuff
<Ursinha> matsubara, nope
* bigjools-afk is now known as bigjools
<stub> mars: I think it was a cock up rather than a deliberate change
<Ursinha> but we have some outstanding oopses that keep happening
<Ursinha> who's on behalf of flacoste today?
<matsubara> Ursinha, yes, those are the ones I meant
<matsubara> Ursinha, mars 
<mars> stub, agreeded, but if so, did it just happen to slip past the dev, and the rc-reviewer?
<stub> https://code.edge.launchpad.net/~stub/launchpad/pending-db-changes/+merge/9188 is also awaiting rc anyway. What I'm more interested in is how come the test suite still passes.
<mars> Ursinha, I'm standing in for flacoste
<matsubara> stub, why the branch that introduced the critical bug wasn't QA'd on staging?
<Ursinha> matsubara, I guess that was a bugs branch
<matsubara> oh, sorry. I thought it was a foundations issue
<mars> stub, has the test suite changed to accommodate open sourcing in some way?  And thus removing those components from the suite?
<Ursinha> matsubara, the missing links are, but which introduced that not
<matsubara> Ursinha, ok.
<stub> mars: I don't know.
<Ursinha> mars, we have an oopses that happens almost everyday
<Ursinha> *an oops
<Ursinha> mars, https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1300XMLP5
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
<Ursinha> mars, once I talked with flacoste about them, and he said that basically that wasn't our problem, iirc :)
<mars> Ursinha, who would be the 'us' in "our problem"?
<Ursinha> mars, foundations/launchpad
<Ursinha> mars, i.e. not a bug
<Ursinha> do you know what could we do to avoid those?
<abentley> Ursinha: is this the private or public XMLRPC server?
<mars> Ursinha, do we have a procedure for dealing with such noise?  The feels like a 404: it is an error in the system state, but not a programmer error.
<Ursinha> I mean, we have a list of oopses that are growing every day because of hanging little things
<Ursinha> mars, that's exactly my question :)
<mars> Ursinha, for this specific issue, I would have to investigate.
<Ursinha> mars, if we could deal differently those oopses, raising 404 instead of that oops, or even if we (me and matsubara) could move them to another section of the report
<matsubara> abentley, private one
<mars> for the larger issue of noise, well, maybe creating a new OOPS category or tag that can filter them out
<stub> matsubara: The branch was rc, so not much QA time. Even if it was qa'd, that doesn't mean anyone would have tested the login or shipit systems which should have been unaffected according to the commit message.
<abentley> matsubara: Well, it kinda is our problem, since we control both ends, then.
<mars> Ursinha, not 404, but one of the other status codes, for "Ill-formed request"
<Ursinha> mars, sorry, bad brain link :)
<abentley> I agree that bad input should not be treated as an OOPS
<matsubara> I don't like moving those oopses to another section.
<Ursinha> matsubara, me neither.
<abentley> But it should be logged *somewhere*.
<matsubara> if the input is bad and we know it, let's not log an oops for it
<Ursinha> mars, could you do some more investigation on this? I'll file a bug, can I?
<matsubara> we used to have a bug for ExpatErrors
* Ursinha looks
<matsubara> was it fixed?
<mars> Ursinha, certainly, please do
<Ursinha> mars, ok, so I'll search for the bug, and file one if not able to find
<matsubara> [action] ursinha do file bug for OOPS-1300XMLP5
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
<MootBot> ACTION received:  ursinha do file bug for OOPS-1300XMLP5
<ubottu> https://lp-oops.canonical.com/oops.py/?oopsid=1300XMLP5
<Ursinha> thanks matsubara
<matsubara> Ursinha, anything else?
<mars> I think the procedure is to turn bad data from a "500: Internal Server Error" into a "400: Bad Request"
<Ursinha> matsubara, yes
<Ursinha> copy and paste fail
<Ursinha> we also have constant UnicodeDecodeErrors and the like
<Ursinha> everyday, and I noticed that they're somehow growing
<Ursinha> is the unicode problem really unsolvable?
<abentley> Ursinha: A bunch of them should disappear as of this rollout.
<Ursinha> abentley, right, but they seem to be everywhere. I'll keep one eye on them, and will report back next meeting
<Ursinha> matsubara, [action] Ursinha to ^
<Ursinha> :P
<Ursinha> mars, this one is hanging for some time: bug 354593
<mars> Ursinha, a unicode error in Python is solvable.  Are they coming from everywhere in the system?
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<abentley> Ursinha: If you're looking for an ultimate solution, it's along the lines of "use Python 3.0".
<matsubara> [action] Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting
<MootBot> ACTION received:  Ursinha to keep one eye on UnicodeDecodeErrors, and will report back next meeting
<Ursinha> abentley, :)
<Ursinha> mars, I'll show you later, we have several oopses and different bugs for them
<Ursinha> 5 digit bugs :)
<Ursinha> mars, can you take a look in that bug, please?
<mars> ok
<Ursinha> using kiko's words: "if they happen more than once a week they are not low priority"
<Ursinha> matsubara, [action] mars to take a look at bug 354593
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<mars> ok, so we know the problem, and we have an ideal solution
<matsubara> [action] mars to take a look at bug 354593
<MootBot> ACTION received:  mars to take a look at bug 354593
<mars> did Francis mention a quick interim solution?
<mars> stub, ^?
<stub> interim solution of what?
<Ursinha> stub, bug 354593
<ubottu> Launchpad bug 354593 in launchpad-foundations "SSO exceptions views need proper branding" [High,Triaged] https://launchpad.net/bugs/354593
<stub> No - I haven't discussed that with him.
<mars> ok
<mars> I'll have a look at the reports, split them into new bugs, try to eliminate the OOPSes
<Ursinha> mars, can you make comments in the bug, if that applies, please?
<stub> It should be possible to register new exception views for that layer, no?
<mars> the branding is a related, but larger issue
<mars> Ursinha, sure
<matsubara> Ursinha, anything else? we need to move on as we only have 11min
<Ursinha> matsubara, no, I'm done for today.
<matsubara> thanks everyone
<Ursinha> thanks mars, stub, abentley and herb
<matsubara> [TOPIC] * Operations report (mthaddon/herb/spm)
<MootBot> New Topic:  * Operations report (mthaddon/herb/spm)
<herb> 2009-07-19 - We had another app server hang during log rotation. Fortunately it seems there has been some progress on bug #287304
<Ursinha> and all the others :)
<ubottu> Launchpad bug 287304 in launchpad-foundations "App Servers: Remove need for restart on logrotation" [High,Incomplete] https://launchpad.net/bugs/287304
<herb> 2009-07-21 - We open sourced! Congratulations to all the developers on a job well done. We saw an uptick in traffic to the codehosting system and, apparently, quite a few new project registrations. Overall the system handled the increased load well.
<herb> 2009-07-22 - Yesterday evening we rolled out 2.2.7. We had some hiccups again but we have bugs filed for those, as was discussed during the critical bugs section.
<herb> That's it from the LOSAs unless there are questions.
<matsubara> thanks herb 
<matsubara> [TOPIC] * DBA report (stub)
<MootBot> New Topic:  * DBA report (stub)
<herb> thanks matsubara
<stub> Database update went painlessly, or at least I haven't heard any tales of pain and misery and nobody woke me up.
<stub> With 2.2.7, we can start pruning unwanted People. We just need to add '--experimental' to garbo-daily.py and maybe tweak the --abort-script argument if the default of 24 hours is too long.
<stub> Nothing else interesting happening.
<matsubara> stub, are you taking care of enabling the pruning script? or have an RT for the losas to do it?
<mthaddon> stub: yep, db updates were pretty quick and painless this time around
<stub> I think salgado might have done that already. I was going to worry about it after reroll and the fires are all out.
<matsubara> [action] matsubara to chase salgado about people pruning script
<MootBot> ACTION received:  matsubara to chase salgado about people pruning script
<matsubara> ok, anything else for stub?
<matsubara> thanks stub 
<matsubara> before I close, just want to let you know that I created the 2.2.8 milestone
<matsubara> during yesterday's TLs meeting, the TL said it would be useful to coordinate work even if we're not doing a release for 2.2.8
<matsubara> so please, since we're not doing the 2.2.8 release, don't let your QA itens otherwise will have a huge backlog for 2.2.9
<matsubara> s/itens/items slip/
<matsubara> I think that's all for today
<matsubara> Thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
<matsubara> #endmeeting 
<MootBot> Meeting finished at 10:43.