1 <matsubara> Welcome to this week's Launchpad Production Meeting. For the next 45 minutes or so, we'll be coordinating the resolution of specific Launchpad bugs and issues.
   2 <Ursinha> erm
   3 <matsubara> [TOPIC] Roll Call 
   4 <herb> me
   5 <Ursinha> me
   6 <mrevell> me
   7 <sinzui> me
   8 <rockstar> we
   9 <matsubara> so who's here today?
  10 <sinzui> I'm still here
  11 <Ursinha> everyone except the bot
  12 <rockstar> me too
  13 <matsubara> lag
  14 <matsubara> flacoste, bigjools: ?
  15 <bigjools> me
  16 <flacoste> me
  17 * henninge (n=henning@i577A8476.versanet.de) has joined #launchpad-meeting
  18 <henninge> me
  19 * stub (n=stub@canonical/launchpad/stub) has joined #launchpad-meeting
  20 <intellectronica> me
  21 <matsubara> all right. everyone is here
  22 <matsubara> [TOPIC] Agenda 
  23 <matsubara>  * Next meeting
  24 <matsubara>  * Actions from last meeting
  25 <matsubara>  * Oops report & Critical Bugs 
  26 <matsubara>  * Operations report (mthaddon/herb/spm)
  27 <matsubara>  * DBA report (DBA contact)
  28 <matsubara> [TOPIC] * Next meeting
  29 <matsubara> so next meeting, same time next week?
  30 <Ursinha> +1
  31 <matsubara> how about removing that section and adding it again only when we won't be having the meeting?
  32 <rockstar> This part of the meeting should only be here if there's something that might make it not at the same time.
  33 <rockstar> :)
  34 <matsubara> :-)
  35 <matsubara> [action] matsubara to remove the next meeting from the agenda and add a note to the MeetingAgenda page that it should be added again when we need to change the time
  36 <flacoste> i'll need to send somebody
  37 <flacoste> as I might not be able to attend (TL meeting)
  38 <bigjools> me too
  39 <flacoste> same for bigjools and sinzui
  40 <sinzui> me too
  41 <matsubara> same here, I'll be attending the TL meeting so Ursinha will likely run it
  42 <Ursinha> yes
  43 <henninge> so there is a reason to discuss the next meeting at each meeting?
  44 <henninge> -> planned attendance
  45 <matsubara> [action] TL (sinzui, flacoste, bigjools) to send people to cover for them due the TL meeting next week
  46 <matsubara> indeed henninge 
  47 <matsubara> so, moving on
  48 <matsubara> [TOPIC] * Actions from last meeting
  49 <matsubara>  * matsubara and Ursinha to QA some of registry team items
  50 <matsubara> ok, we did that.
  51 <sinzui> Thanks very much
  52 <matsubara> so, if any of the teams need a hand testing stuff, talk to us!
  53 <matsubara> sinzui, my pleasure :-)
  54 <matsubara>  * Oops report & Critical Bugs 
  55 <matsubara> Today oops report is about bugs 319173, 318894, OOPS-1116EC265, OOPS-1113EB6
  56 <ubottu> Launchpad bug 319173 in launchpad "zip files cannot be uploaded to a release" [Undecided,Incomplete] https://launchpad.net/bugs/319173
  57 <ubottu> Launchpad bug 318894 in launchpad-foundations "oops when responding to merge proposal review via email" [Undecided,New] https://launchpad.net/bugs/318894
  58 <matsubara> so the first one, which I'll move to -registry I need to try to reproduce on windows
  59 <flacoste> herb: can you check that buildbot-poll script is working? buildbot has blessed a lot of revisions that weren't pushed to stable
  60 <matsubara> Im working with the reporter to find out the cause and will add more info once I have it
  61 <sinzui> matsubara: I think we limit the upload to tar.gz, changelof, and .txt
  62 <matsubara> sinzui, who in your team should be assigned to fix that one?
  63 <herb> flacoste: will check after the meeting, yes
  64 <sinzui> matsubara: is it a bug?
  65 <matsubara> sinzui, it works on firefox for linux. it doesn't matter the file extension
  66 <matsubara> sinzui, the thing is, according to the reporter, it oopses when he tries to upload while using FF on windows 
  67 <sinzui> matsubara: I'll give it to my favorite bug assassin--salgado
  68 <matsubara> sinzui, ok, once I have more info I'll coordinate with salgado. thanks sinzui 
  69 <matsubara> the other two oopses are for lp-bzr team
  70 <matsubara> [action] matsubara to file bugs for OOPS-1116EC265, OOPS-1113EB6
  71 <matsubara> rockstar, can you help me out with those two oopses after the meeting? need to find a way to reproduce them
  72 <rockstar> matsubara, okay.
  73 <matsubara> flacoste, news about the -foundations one?
  74 <flacoste> matsubara: looking into it, but unless it is happening a lot, i don't see it as a high priority
  75 <matsubara> flacoste, barry's been bitten a lot by that one, IIRC
  76 <flacoste> that's what you get for using proprietary email applications!
  77 <matsubara> flacoste, shall I close it as invalid with that as the rationale? :-)
  78 <flacoste> i'll take care of it :-)
  79 <matsubara> flacoste, thanks!
  80 <matsubara> for critical bugs, we have 3. all of them fix committed.
  81 <matsubara> good job everyone!
  82 <matsubara> moving on
  83 <matsubara>  * Operations report (mthaddon/herb/spm)
  84 <herb> 2009-01-16 - Friday - We cherry picked r7586 to lpnet* to fix bug #315103
  85 <ubottu> Launchpad bug 315103 in malone "When consuming a token, Launchpad tries to update already-consumed tokens" [High,Fix committed] https://launchpad.net/bugs/315103
  86 <herb> 2009-01-20 - Tuesday - We cherry picked r7588 to lpnet* to fix bug #317241
  87 <ubottu> Launchpad bug 317241 in launchpad-foundations "Creating a launchpad account in the process of an OpenID authentication returns a failure to the RP" [Critical,Fix committed] https://launchpad.net/bugs/317241
  88 <herb> Bug #156453 and #118625 continue to be a problem requiring us to restart codebrowse at least daily.
  89 <ubottu> Launchpad bug 156453 in loggerhead "production loggerhead branch leaks memory" [Critical,In progress] https://launchpad.net/bugs/156453
  90 <herb> As noted by mthaddon and spm via email to the list, we've been seeing higher than normal DB load this week. We also experienced some issues with app servers taking too long to respond to requests (anywhere from 15-30 seconds). We suspect the two are related in some way but don't yet have any hard evidence to back this up.
  91 <ubottu> Launchpad bug 118625 in launchpad-bazaar "codebrowse sometimes hangs" [High,Triaged] https://launchpad.net/bugs/118625
  92 <Ursinha> hm, this explains the timeouts
  93 <matsubara> linkchecker was disabled last friday, but that didn't seem to make any difference on the staging DB load
  94 <herb> well. that's it from us unless there are questions.
  95 <matsubara> rockstar, how's the codebrowser bug going?
  96 <stub> The staging rebuild is the killer - it chews up an entire CPU and swaps important information out of RAM for its run, which is about 8 hours of the day IIRC.
  97 <rockstar> matsubara, we know no more this week than we did last week.
  98 <matsubara> rockstar, :-(
  99 <rockstar> jml and poolie are at the TL sprint early to work on it.
 100 <matsubara> that's great news
 101 <matsubara> stub, any way to optimize that?
 102 <stub> It already is optimized.
 103 <stub> We would have to start dropping features, which makes it less useful.
 104 <flacoste> or get a new box
 105 <flacoste> kiko is on it
 106 <flacoste> gpg: Erreur de CRC; E7341E - DC3E73
 107 <flacoste> gpg: aucune signature trouvée
 108 <flacoste> gpg: impossible de vérifier la signature.
 109 <flacoste> ignore those gpg thing
 110 <matsubara> hehe that's barry's broken signature :-)
 111 <matsubara> all right. thanks stub, flacoste, herb and rockstar 
 112 <matsubara> moving on
 113 <matsubara> [TOPIC] * DBA report (DBA contact)
 114 <stub> Staging rebuilds are getting dangerously close to filling up their disk partition again. Avoiding this looming problem involves moving staging to new hardware, or bringing down the databases on Chokecherry while we repartition the disks.
 115 <stub> We have been getting some load spikes again, causing the losas to rewire the appservers to distribute db load a little more evenly. In the past, these have been bots request spamming. I don't think anyone has identified the trigger this time yet. My money is on a bot crawling an expensive part of the site, or the load of the staging rebuild causing trouble, or a combination of the two.
 116 <stub> I've suggested shutting down staging and the staging rebuild next time it happens to reduce the possible variables.
 117 <stub> DB patches for this cycle have all landed except for one of mine still being worked on (severing relationships between the future authentication replication set and the main db), and a late request from Salgado (caching the total download count on LibraryFileAlias, which I'll probably defer until next cycle rather than make a hasty decision).
 118 * e-jat (n=fenris@ubuntu/member/fenris-) has joined #launchpad-meeting
 119 <matsubara> all right, questions for stub?
 120 <matsubara> thanks stub 
 121 <matsubara> what do you think of officially make this meeting 30 min rather than 45?
 122 <Ursinha> matsubara, we in average use 40-45 mins
 123 <Ursinha> do you think it's needed?
 124 <Ursinha> to reduce the time, that is
 125 <matsubara> Ursinha, do we? I thought we used less than 30.
 126 <matsubara> but let's keep it 40-45 then
 127 <Ursinha> AFAIR
 128 <matsubara> anyway, thank you all for attending this week's Launchpad Production Meeting. See the channel topic for the location of the logs. 
 129 <matsubara> #endmeeting