Moving to newer Python versions

With Lucid fast approaching, efforts to actually migrate Launchpad to Python 2.6 are getting under way.

Relevant tagged bugs: https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade

Branches

• https://code.launchpad.net/~launchpad-committers/launchpad/python2.6 -- collaboration branch created by maxb based on salgado's personal branch.

• https://code.launchpad.net/~salgado/shipit/shipit-python2.6

Issues that need revisiting

• The failure when setting up the GoogleServiceLayer seems to be a consequence of the server answering with a errno.ECONNABORTED when it's not yet ready, so salgado has just changed the wait_for_service function to treat that error the same way it does for errno.ECONNREFUSED.

• The test hang is because of the change done on bug 349316, so salgado has just commented out that addCleanup() line for now, to see if we can get a full test run.

• A multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests would fail merely because of unexpected DeprecationWarnings, so we've had to jump through multiple hoops all over the place to suppress these. Ultimately we need a plan for removing these suppressions, because we want to know about deprecation long-term.

  • It looks like an attainable goal to get all of our dependencies fixed to not raise DeprecationWarnings. The ones needing work are: Twisted, PyCrypto, Ampoule.

Setup caveats

• The extension modules in your sourcecode directory will have been built for Python 2.5 and these will break for your 2.6 branch. The simple thing to do is to move your old sourcecode aside and run utilities/rocketfuel-get to get it again with Python 2.6. A more complex but useful thing to do to manage sourcecode directories for multiple python versions is:

  cd ~/launchpad/lp-sourcedeps
  mkdir 2.6
  cd 2.6
  ln -s ../download-cache
  ln -s ../eggs
  mkdir sourcecode
  cd sourcecode
  for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done

  and now point utilities/link-external-sourcecode to ../../lp-sourcedeps/2.6 - if you do that you'll need to be aware that rocketfuel-get won't know about it, and your extra directories will require occasional manual updating.

Known test failures and notes thereon

These results assume you:

• Use the lp:~launchpad-committers/launchpad/python2.6 branch
• Have the following not-yet-landed branches merged:
  • None at the moment.

• Do not have python-tdb installed, which otherwise causes bzr-hg and bzr-git to behave in ways the tests do not expect.

BaseLayer: OK (281 tests)

MemcachedLayer: OK (8 tests)

LibrarianLayer: OK (9 tests)

DatabaseLayer: OK (46 tests)

LaunchpadLayer: OK (24 tests)

ZopelessLayer: OK (20 tests)

LaunchpadScriptLayer: OK (85 tests)

LaunchpadZopelessLayer: OK

FunctionalLayer: OK (174 tests)

LaunchpadFunctionalLayer

• valid_absolute_url (canonical.launchpad.validators.url) -- Python 2.6's urlparse disregards the uses_netloc list when parsing urls and treats anything beginning with // as netloc based. This makes maxb and wgrant say WTF!?! The untested valid_absolute_url DB constraint is also affected!

  • This behavior is expected and is in compliance with rfc3986, so we just need to change/remove our test. -- Salgado

AppServerLayer: OK (143 tests)

DatabaseFunctionalLayer: OK

GoogleLaunchpadFunctionalLayer: OK (72 tests)

PageTestLayer: OK (6335 tests)

TwistedLayer: OK (235 tests)

TwistedLaunchpadZopelessLayer: OK (16 tests)

TwistedAppServerLayer

• test_import_git (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegration)

  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 371, in _runCallbacks
      self.result = callback(self.result, *args, **kw)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 674, in assertImported
      self.assertCodeImportResultCreated(code_import)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 657, in assertCodeImportResultCreated
      self.assertEqual(result.status, CodeImportResultStatus.SUCCESS)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 287, in failUnlessEqual
      % (msg, pformat(first), pformat(second)))
  FailTest: not equal:
  a = <DBItem CodeImportResultStatus.FAILURE, (200) Failure>
  b = <DBItem CodeImportResultStatus.SUCCESS, (100) Success>

• test_import_git (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegrationScript)

  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 371, in _runCallbacks
      self.result = callback(self.result, *args, **kw)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 674, in assertImported
      self.assertCodeImportResultCreated(code_import)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 657, in assertCodeImportResultCreated
      self.assertEqual(result.status, CodeImportResultStatus.SUCCESS)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 287, in failUnlessEqual
      % (msg, pformat(first), pformat(second)))
  FailTest: not equal:
  a = <DBItem CodeImportResultStatus.FAILURE, (200) Failure>
  b = <DBItem CodeImportResultStatus.SUCCESS, (100) Success>

ZopelessAppServerLayer: OK

ZopelessDatabaseLayer: OK (259 tests)

BugsWindmillLayer: OK (13 tests)

CodeWindmillLayer: OK (12 tests)

ForeignBranchPluginLayer

• test_import (lp.codehosting.codeimport.tests.test_worker.TestGitImport)

  Traceback (most recent call last):
  _StringException: Text attachment: log
  ------------
  70.600  creating repository in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/.
  70.606  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x443a610> in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/
  71.339  creating repository in file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_import/work/bazaar_store/00000001/.bzr/.
  71.344  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x443a610> in file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_import/work/bazaar_store/00000001/.bzr/
  71.354  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
  71.354  fetch up to rev {git-v1:7dc6761613187e08d6f0344f0f4e8d51a3ebd440}
  71.376  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
  71.376  fetch up to rev {None}
  ------------
  Text attachment: traceback
  ------------
  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
      return fn(*args)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
      testMethod()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 754, in test_import
      worker.run()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 434, in run
      return self._doImport()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 574, in _doImport
      self.pushBazaarBranch(bazaar_branch)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 639, in pushBazaarBranch
      create_tarball(git_db_dir, local_name)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tarball.py", line 46, in create_tarball
      raise NotADirectory(directory)
  NotADirectory: /var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/git is not a directory.

• test_import_script (lp.codehosting.codeimport.tests.test_worker.TestGitImport)

  Failure in test test_import_script (lp.codehosting.codeimport.tests.test_worker.TestGitImport)
  Traceback (most recent call last):
  _StringException: Text attachment: log
  ------------
  
  ------------
  Text attachment: traceback
  ------------
  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
      return fn(*args)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
      testMethod()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 793, in test_import_script
      self.assertEqual(retcode, 0)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/bzr-2.1.0-py2.6-linux-x86_64.egg/bzrlib/tests/__init__.py", line 1095, in assertEqual
      pformat(a), pformat(b)))
  AssertionError: not equal:
  a = 1
  b = 0

• test_partial (lp.codehosting.codeimport.tests.test_worker.TestGitImport)

  Error in test test_partial (lp.codehosting.codeimport.tests.test_worker.TestGitImport)
  Traceback (most recent call last):
  _StringException: Text attachment: log
  ------------
  74.796  creating repository in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/.
  74.801  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x443a610> in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/
  75.478  creating repository in file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_partial/work/bazaar_store/00000001/.bzr/.
  75.483  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x443a610> in file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_partial/work/bazaar_store/00000001/.bzr/
  75.493  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_partial/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
  75.493  fetch up to rev {git-v1:74e769bfafeb8e54cb3a5fa31c3a7b91dfbf6a58}
  75.514  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_partial/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
  75.515  fetch up to rev {None}
  ------------
  Text attachment: traceback
  ------------
  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
      return fn(*args)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
      testMethod()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 979, in test_partial
      CodeImportWorkerExitCode.SUCCESS_PARTIAL, worker.run())
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 434, in run
      return self._doImport()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 574, in _doImport
      self.pushBazaarBranch(bazaar_branch)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 639, in pushBazaarBranch
      create_tarball(git_db_dir, local_name)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tarball.py", line 46, in create_tarball
      raise NotADirectory(directory)
  NotADirectory: /var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/git is not a directory.
  ------------

• test_script_exit_codes (lp.codehosting.codeimport.tests.test_worker.TestGitImport)

  Failure in test test_script_exit_codes (lp.codehosting.codeimport.tests.test_worker.TestGitImport)
  Traceback (most recent call last):
  _StringException: Text attachment: log
  ------------
  
  ------------
  Text attachment: traceback
  ------------
  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
      return fn(*args)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
      testMethod()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 831, in test_script_exit_codes
      self.assertEqual(retcode, CodeImportWorkerExitCode.SUCCESS)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/bzr-2.1.0-py2.6-linux-x86_64.egg/bzrlib/tests/__init__.py", line 1095, in assertEqual
      pformat(a), pformat(b)))
  AssertionError: not equal:
  a = 1
  b = 0
  
  ------------

• test_sync (lp.codehosting.codeimport.tests.test_worker.TestGitImport)

  Error in test test_sync (lp.codehosting.codeimport.tests.test_worker.TestGitImport)
  Traceback (most recent call last):
  _StringException: Text attachment: log
  ------------
  78.882  creating repository in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/.
  78.887  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x443a610> in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/
  79.566  creating repository in file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_sync/work/bazaar_store/00000001/.bzr/.
  79.571  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x443a610> in file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_sync/work/bazaar_store/00000001/.bzr/
  79.582  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_sync/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
  79.582  fetch up to rev {git-v1:7c51b07ab1942421857ccd7a3ece5e45b5cb12fa}
  79.603  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-alI4LS.tmp/lp.codehosting.codeimport.tests.test_worker.TestGitImport.test_sync/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
  79.603  fetch up to rev {None}
  ------------
  Text attachment: traceback
  ------------
  Traceback (most recent call last):
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
      return fn(*args)
    File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
      testMethod()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 763, in test_sync
      worker.run()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 434, in run
      return self._doImport()
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 574, in _doImport
      self.pushBazaarBranch(bazaar_branch)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/worker.py", line 639, in pushBazaarBranch
      create_tarball(git_db_dir, local_name)
    File "/home/salgado/devel/launchpad/python2.6/lib/lp/codehosting/codeimport/tarball.py", line 46, in create_tarball
      raise NotADirectory(directory)
  NotADirectory: /var/tmp/codeimport/data/worker-for-branch-1/bzr_branch/.bzr/repository/git is not a directory.
  ------------

SSHServerLayer: OK (46 tests)

RegistryWindmillLayer: OK (17 tests)

SoyuzWindmillLayer: OK (2 tests)

TestZopeTestInSubProcessLayer: OK (1 test)

TranslationsWindmillLayer: OK (7 tests)

UnitTests: OK

Shipit: OK

MailmanLayer: OK