Moving to newer Python versions

The ultimate goal is to migrate Launchpad to Python 2.6. Rather than go straight to Python 2.6, we are focusing on Python 2.5 first as a stepping stone. People involved in this effort: maxb, gary, salgado, barry, simono. We are treating this work as a sprint; come join us on #launchpad-sprint on freenode.

Relevant tagged bugs: https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade

Branches

• https://code.launchpad.net/~launchpad-dev/launchpad/python-migration is the master branch tracking the migration to Python 2.5. This was branched off of devel. You should submit merge proposals for your branches into this one. gary, salgado, and barry will review all branches and we'll use a lightweight review process to manage this (simple cover letters, 5-minute reviews). Once approved, you can land your change by just pushing it into python-migration; there is no pqm gateway.

• https://code.launchpad.net/~launchpad-dev/launchpad/ztk-2.5 is the python-migration branch with the latest Zope Toolkit updates, still on Python 2.5.

• https://code.launchpad.net/~launchpad-dev/launchpad/python-migration2.6 is the ztk-2.5 branch with changes to get us on Python 2.6.

• https://code.edge.launchpad.net/~salgado/shipit/py2.5 is the shipit branch that make it possible to run shipit under 2.5

• https://code.edge.launchpad.net/~salgado/canonical-identity-provider/ztk-2.5 has a test fix, needed when we change to ztk on 2.5

In general, we'll merge from devel -> python-migration -> ztk-2.5 -> python-migration2.6, but we're keeping them separate in case something prevents us from making it all the way to 2.6.

Things you'll need to do

• The Makefiles of pygettextpo & pygpgme (in sourcecode/) make direct reference to Python 2.4, you'll need to amend them appropriately.

Current status of the Python 2.5 effort

It passes all of its tests!

Known/Open issues

Bustimications

• The extension modules in your sourcecode directory will have been built for Python 2.4 and these will break until your 2.5 branch. The simple thing to do is to move your old sourcecode aside and run utilities/rocketfuel-get to get it again with Python 2.5. A more complex but useful thing to do to manage sourcecode directories for multiple python versions is:

  cd ~/launchpad/lp-sourcedeps
  mkdir 2.5
  cd 2.5
  ln -s ../download-cache
  ln -s ../eggs
  mkdir sourcecode
  cd sourcecode
  for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done

  and now point utilities/link-external-sourcecode to ../../lp-sourcedeps/2.5 - if you do that you'll need to be aware that rocketfuel-get won't know about it, and your extra directories will require occasional manual updating.

• The pytz source package has a mysterious problem, hence we have eggs in download-cache.

• _lsprof is in sourcecode because Python 2.4 doesn't come with it. Python 2.5 and 2.6 do though, so we'll be able to remove it from sourcecode once we upgrade. Except, it's not quite as simple as that. Currently we're in a bizare mixed state where the _lsprof.so symlink has been removed, but the cProfile.py one has not been. Furthermore, pstats2.py is also provided by our lsprof, and there's no pstats2 in Python - there's a pstats, though.

Cleanup potential

• Eventually we should get rid of all the scripts hardcoded /usr/bin/pythonX.Y and use buildout's shebang munging.

• scripts/branch-rewrite.py uses /usr/bin/pythonX.Y -u. We should figure out if this is significant and why this script has to be different from other scripts.

• scripts/ftpmaster-tools/_syncorigins.py uses /usr/bin/env pythonX.Y... why?!

• cronscripts/mirror-prober.sh and cronscripts/nightly.sh are invoking python scripts with explicit interpreters. Why are they not using the shebangs? This leads to versions being defined in two places.

Python 2.5 w/Zope Toolkit

This is a branch off of python-migration which uses Python 2.5 and the Zope Toolkit.

test failures

• lib/canonical/shipit/tests/../doc/shipit.txt
  • Salgado has a branch that fixes this
• lib/canonical/signon/browser/tests/loginservice-unauthorized-view.txt
  • Salgado has a branch that fixes this

Python 2.6

This is the ztk-2.5 branch, converted to use Python 2.6.

Also, a multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests will fail merely because of unexpected DeprecationWarnings (but see below).

Getting started

• Apply this patch and re-run bin/buildout to temporarily suppress the turning of warnings into errors. There are DeprecationErrors coming from Twisted that we can't seem to turn off any other way.

make run failures

Tests with failures:
   test_import_script (lp.codehosting.codeimport.tests.test_worker.TestGitImport)
   testLibrarianRunning (canonical.testing.ftests.test_layers.DatabaseTestCase)
   testLibrarianRunning (canonical.testing.ftests.test_layers.ZopelessTestCase)
   lib/canonical/launchpad/ftests/../doc/old-testing.txt
   testLaunchpadDbAvailable (canonical.testing.ftests.test_layers.FunctionalTestCase)
   testLibrarianRunning (canonical.testing.ftests.test_layers.FunctionalTestCase)
   lib/canonical/shipit/tests/../doc/shipit-process-requests.txt
   lib/canonical/signon/browser/tests/loginservice-unauthorized-view.txt
   lp.codehosting.tests.test_rewrite.TestBranchRewriterScript.test_script
   lib/lp/registry/browser/tests/gpg-views.txt
   lib/lp/registry/tests/../doc/cache-country-mirrors.txt
   lib/lp/registry/tests/../doc/convert-person-to-team.txt
   lib/lp/registry/tests/../doc/teammembership.txt
   BuilddSlaveTestSetup (canonical.buildd.tests.harness)
   make_test_tarball_2 (canonical.launchpad.tests.test_helpers)
   shortlist (canonical.launchpad.helpers)
   shortlist (canonical.launchpad.helpers)
   _enable_bzr_ssh_in_urlparse (canonical.launchpad.webapp.url)
   urlparse (canonical.launchpad.webapp.url)
   urlsplit (canonical.launchpad.webapp.url)
   /home/barry/projects/launchpad/python-migration2.6/lib/canonical/lazr/tests/../doc/timeout.txt
   LibrarianTestSetup (canonical.librarian.ftests.harness)
   ZecaTestSetup (canonical.zeca.ftests.harness)
   test_successful_start_then_stop (lp.codehosting.tests.test_lpserve.TestLaunchpadServe)
   test_unexpected_error_logs_oops (lp.codehosting.tests.test_lpserve.TestLaunchpadServe)
   lp.services.scripts.tests.test_all_scripts.ScriptsTestCase.test_scripts
   lib/lp/translations/tests/../doc/remove-upstream-translations-script.txt
Total: 6208 tests, 27 failures, 27 errors in 28 minutes 46.400 seconds.