Moving to newer Python versions

With Lucid fast approaching, efforts to actually migrate Launchpad to Python 2.6 are getting under way.

The current devel branch has its test suite passing on python2.6, provided you change the places where python2.5 is hard-coded (scripts and Makefiles). Salgado also has a branch which changes scripts and Makefiles to use the default system python, which means that branch can be used to run the full LP test suite on either Hardy or Lucid.

The use-default-python branch above requires this pygettextpo branch.

It's not yet decided whether we're going to proceed with this approach of using the default system python or a different one, using something like '/usr/bin/lp-python' (which would be created by launchpad-dependencies). Whatever approach we take, we'll also need to update this shipit branch.

Blockers

Checkwatches doesn't work with python2.6. That should be fixed before we switch to python2.6 on production.

Branches

• https://code.launchpad.net/~launchpad-committers/launchpad/python2.6 -- collaboration branch created by maxb based on salgado's personal branch.

• https://code.launchpad.net/~salgado/shipit/shipit-python2.6

Issues that need revisiting

• The failure when setting up the GoogleServiceLayer seems to be a consequence of the server answering with a errno.ECONNABORTED when it's not yet ready, so salgado has just changed the wait_for_service function to treat that error the same way it does for errno.ECONNREFUSED.

• The test hang is because of the change done on bug 349316, so salgado has just commented out that addCleanup() line for now, to see if we can get a full test run.

• A multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests would fail merely because of unexpected DeprecationWarnings, so we've had to jump through multiple hoops all over the place to suppress these. Ultimately we need a plan for removing these suppressions, because we want to know about deprecation long-term.

  • It looks like an attainable goal to get all of our dependencies fixed to not raise DeprecationWarnings. The ones needing work are: Twisted, PyCrypto, Ampoule.

• Because of upstream changes (http://bugs.python.org/issue7904), the effect of upgrading the database to Python 2.6 will significantly change the behaviour of the valid_absolute_url constraint stored procedure to accept any scheme in URLs. We should (1) confirm whether this is OK, and (2) plan to remove all the unnecessary fiddling with uses_netloc is this is the case.

Setup caveats

• The extension modules in your sourcecode directory will have been built for Python 2.5 and these will break for your 2.6 branch. The simple thing to do is to move your old sourcecode aside and run utilities/rocketfuel-get to get it again with Python 2.6. A more complex but useful thing to do to manage sourcecode directories for multiple python versions is:

  cd ~/launchpad/lp-sourcedeps
  mkdir 2.6
  cd 2.6
  ln -s ../download-cache
  ln -s ../eggs
  mkdir sourcecode
  cd sourcecode
  for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done

  and now point utilities/link-external-sourcecode to ../../lp-sourcedeps/2.6 - if you do that you'll need to be aware that rocketfuel-get won't know about it, and your extra directories will require occasional manual updating.

Known test failures and notes thereon

These results assume you:

• Use the lp:~launchpad-committers/launchpad/python2.6 branch
• Have the following not-yet-landed branches merged:
  • None at the moment.

• Do not have python-tdb installed, which otherwise causes bzr-hg and bzr-git to behave in ways the tests do not expect.

BaseLayer: OK (281 tests)

MemcachedLayer: OK (8 tests)

LibrarianLayer: OK (9 tests)

DatabaseLayer: OK (46 tests)

LaunchpadLayer: OK (24 tests)

ZopelessLayer: OK (20 tests)

LaunchpadScriptLayer: OK (85 tests)

LaunchpadZopelessLayer: OK

FunctionalLayer: OK (174 tests)

LaunchpadFunctionalLayer: OK

AppServerLayer: OK (143 tests)

DatabaseFunctionalLayer: OK

GoogleLaunchpadFunctionalLayer: OK (72 tests)

PageTestLayer: OK (6335 tests)

TwistedLayer: OK (235 tests)

TwistedLaunchpadZopelessLayer: OK (16 tests)

TwistedAppServerLayer: OK

ZopelessAppServerLayer: OK

ZopelessDatabaseLayer: OK (259 tests)

BugsWindmillLayer: OK (13 tests)

CodeWindmillLayer: OK (12 tests)

ForeignBranchPluginLayer: OK

SSHServerLayer: OK (46 tests)

RegistryWindmillLayer: OK (17 tests)

SoyuzWindmillLayer: OK (2 tests)

TestZopeTestInSubProcessLayer: OK (1 test)

TranslationsWindmillLayer: OK (7 tests)

UnitTests: OK

Shipit: OK

MailmanLayer: OK