Differences between revisions 1 and 180 (spanning 179 versions)

Moving to newer Python versions

Contents

Moving to newer Python versions

With Lucid fast approaching, efforts to actually migrate Launchpad to Python 2.6 are getting under way.

Relevant tagged bugs: https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade

Branches

https://code.launchpad.net/~launchpad-committers/launchpad/python2.6 -- collaboration branch created by maxb based on salgado's personal branch.
https://code.launchpad.net/~salgado/shipit/shipit-python2.6

Issues that need revisiting

The failure when setting up the GoogleServiceLayer seems to be a consequence of the server answering with a errno.ECONNABORTED when it's not yet ready, so salgado has just changed the wait_for_service function to treat that error the same way it does for errno.ECONNREFUSED.
The test hang is because of the change done on bug 349316, so salgado has just commented out that addCleanup() line for now, to see if we can get a full test run.
A multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests would fail merely because of unexpected DeprecationWarnings, so we've had to jump through multiple hoops all over the place to suppress these. Ultimately we need a plan for removing these suppressions, because we want to know about deprecation long-term.
- It looks like an attainable goal to get all of our dependencies fixed to not raise DeprecationWarnings. The ones needing work are: Twisted, PyCrypto, Ampoule.

Setup caveats

The extension modules in your sourcecode directory will have been built for Python 2.5 and these will break for your 2.6 branch. The simple thing to do is to move your old sourcecode aside and run utilities/rocketfuel-get to get it again with Python 2.6. A more complex but useful thing to do to manage sourcecode directories for multiple python versions is:
```
cd ~/launchpad/lp-sourcedeps
mkdir 2.6
cd 2.6
ln -s ../download-cache
ln -s ../eggs
mkdir sourcecode
cd sourcecode
for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done
```
and now point utilities/link-external-sourcecode to ../../lp-sourcedeps/2.6 - if you do that you'll need to be aware that rocketfuel-get won't know about it, and your extra directories will require occasional manual updating.

Known test failures and notes thereon

These results assume you:

Use the lp:~launchpad-committers/launchpad/python2.6 branch
Have the following not-yet-landed branches merged:
- lp:~wgrant/launchpad/emailauthentication.txt-2.6-fix
Do not have python-tdb installed, which otherwise causes bzr-hg and bzr-git to behave in ways the tests do not expect.

BaseLayer: OK (281 tests)

MemcachedLayer: OK (8 tests)

LibrarianLayer: OK (9 tests)

DatabaseLayer: OK (46 tests)

LaunchpadLayer: OK (24 tests)

ZopelessLayer: OK (20 tests)

LaunchpadScriptLayer: OK (85 tests)

LaunchpadZopelessLayer

test_invalid_signature (canonical.launchpad.mail.tests.test_incoming.TestIncoming)

Traceback (most recent call last):
_StringException: Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
    return fn(*args)
  File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
    testMethod()
  File "/home/salgado/devel/launchpad/python2.6/lib/canonical/launchpad/mail/tests/test_incoming.py", line 44, in test_invalid_signature
    self.assertIs(None, current_oops)
  File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 208, in assertIs
    expected is observed, '%r is not %r' % (expected, observed))
  File "/usr/lib/python2.6/unittest.py", line 325, in failUnless
    if not expr: raise self.failureException, msg
AssertionError: None is not <ErrorReport OOPS-1572TEMAIL1 DeprecationWarning: BaseException.message has been deprecated as of Python 2.6>

FunctionalLayer: OK (174 tests)

LaunchpadFunctionalLayer

valid_absolute_url (canonical.launchpad.validators.url) -- Python 2.6's urlparse disregards the uses_netloc list when parsing urls and treats anything beginning with // as netloc based. This makes maxb and wgrant say WTF!?! The untested valid_absolute_url DB constraint is also affected!
- This behavior is expected and is in compliance with rfc3986, so we just need to change/remove our test. -- Salgado

AppServerLayer: OK (143 tests)

DatabaseFunctionalLayer

lib/lp/registry/tests/../doc/gpg-signatures.txt

Failed doctest test for gpg-signatures.txt
  File "lib/lp/registry/tests/../doc/gpg-signatures.txt", line 0

----------------------------------------------------------------------
File "lib/lp/registry/tests/../doc/gpg-signatures.txt", line 138, in gpg-signatures.txt
Failed example:
    master_sig = gpghandler.getVerifiedSignature(content)
Differences (ndiff with -expected +actual):
      Traceback (most recent call last):
    - ...
    - GPGVerificationError...
    +   File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/zope.testing-3.9.4-py2.6.egg/zope/testing/doctest/__init__.py", line 1355, in __run
    +     compileflags, 1) in test.globs
    +   File "<doctest gpg-signatures.txt[line 138, example 25]>", line 1, in <module>
    +   File "/home/salgado/devel/launchpad/python2.6/lib/canonical/launchpad/utilities/gpghandler.py", line 173, in getVerifiedSignature
    +     raise GPGVerificationError(e.message)
    + DeprecationWarning: BaseException.message has been deprecated as of Python 2.6

-  ⇤ ← Revision 1 as of 2009-10-09 17:47:27 → 
  Size: 6491
  Editor: barry
  Comment:
+   ← Revision 180 as of 2010-04-22 12:40:55 → ⇥
  Size: 6628
  Editor: salgado
  Comment:
-Deletions are marked like this.
+Additions are marked like this.
 Line 3:
-The ultimate goal is to migrate Launchpad to Python 2.6. Rather than go straight to Python 2.6, we are focusing on Python 2.5 first as a stepping stone.  People involved in this effort: maxb, gary, salgado, barry
+<<TableOfContents>>
 Line 5:
-https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade
+With Lucid fast approaching, efforts to actually migrate Launchpad to Python 2.6 are getting under way.
 Line 7:
-== Interesting branches ==
+Relevant tagged bugs: https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade
 Line 9:
- * https://code.launchpad.net/~maxb/launchpad/python2.5 tracks `lp:launchpad/stable` (or occasionally `devel`), on a manual basis, and acts as a one-stop-shop for something that people can check out and build using Python 2.5
+== Branches ==
 Line 11:
- * https://code.launchpad.net/~gary/launchpad/py25 is gary's branch tracking the upgrade.
+ * https://code.launchpad.net/~launchpad-committers/launchpad/python2.6 -- collaboration branch created by maxb based on salgado's personal branch.
 * https://code.launchpad.net/~salgado/shipit/shipit-python2.6
-Line 13:
+Line 14:
- * ''add yours here''
+== Issues that need revisiting ==
-Line 15:
+Line 16:
-== Launchpad code itself ==
+ * The failure when setting up the GoogleServiceLayer seems to be a consequence of the server answering with a errno.ECONNABORTED when it's not yet ready, so salgado has just changed the wait_for_service function to treat that error the same way it does for errno.ECONNREFUSED.
 * The test hang is because of the change done on bug Bug:349316, so salgado has just commented out that addCleanup() line for now, to see if we can get a full test run.
 * A multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests would fail merely because of unexpected !DeprecationWarnings, so we've had to jump through multiple hoops all over the place to suppress these. Ultimately we need a plan for removing these suppressions, because we ''want'' to know about deprecation long-term.
  * It looks like an attainable goal to get all of our dependencies fixed to not raise DeprecationWarnings. The ones needing work are: Twisted, PyCrypto, Ampoule.
-Line 17:
+Line 21:
-Remaining changes in `python2.5` branch:
 * Changed `PYTHON_VERSION` in Makefiles
 * Changed shebang in `cronscripts/*`, `scripts/*`,
+== Setup caveats ==
 * The extension modules in your `sourcecode` directory will have been built for Python 2.5 and these will break for your 2.6 branch.  The simple thing to do is to move your old `sourcecode` aside and run `utilities/rocketfuel-get` to get it again with Python 2.6. A more complex but useful thing to do to manage `sourcecode` directories for multiple python versions is: {{{
cd ~/launchpad/lp-sourcedeps
mkdir 2.6
cd 2.6
ln -s ../download-cache
ln -s ../eggs
mkdir sourcecode
cd sourcecode
for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done
}}} and now point `utilities/link-external-sourcecode` to `../../lp-sourcedeps/2.6` - if you do that you'll need to be aware that `rocketfuel-get` won't know about it, and your extra directories will require occasional manual updating.
-Line 21:
+Line 33:
-== Known issues ==
+== Known test failures and notes thereon ==
These results assume you:
 * Use the lp:~launchpad-committers/launchpad/python2.6 branch
 * Have the following not-yet-landed branches merged:
  * lp:~wgrant/launchpad/emailauthentication.txt-2.6-fix
 * Do '''not''' have python-tdb installed, which otherwise causes bzr-hg and bzr-git to behave in ways the tests do not expect.
-Line 23:
+Line 40:
- * The pytz source package has a [[https://launchpad.net/bugs/438634|mysterious problem]].  The work around is to [[http://pypi.python.org/packages/2.5/p/pytz/pytz-2009l-py2.5.egg#md5=b41c8cacb5f4a8a64db05afbdf3ef9ef|download the 2.5 egg]] and put that in your `download-cache/dist` directory.
+=== BaseLayer: OK (281 tests) ===
=== MemcachedLayer: OK (8 tests) ===
=== LibrarianLayer: OK (9 tests) ===
=== DatabaseLayer: OK (46 tests) ===
=== LaunchpadLayer: OK (24 tests) ===
=== ZopelessLayer: OK (20 tests) ===
=== LaunchpadScriptLayer: OK (85 tests) ===
=== LaunchpadZopelessLayer ===
 * test_invalid_signature (canonical.launchpad.mail.tests.test_incoming.TestIncoming){{{
Traceback (most recent call last):
_StringException: Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
    return fn(*args)
  File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
    testMethod()
  File "/home/salgado/devel/launchpad/python2.6/lib/canonical/launchpad/mail/tests/test_incoming.py", line 44, in test_invalid_signature
    self.assertIs(None, current_oops)
  File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 208, in assertIs
    expected is observed, '%r is not %r' % (expected, observed))
  File "/usr/lib/python2.6/unittest.py", line 325, in failUnless
    if not expr: raise self.failureException, msg
AssertionError: None is not <ErrorReport OOPS-1572TEMAIL1 DeprecationWarning: BaseException.message has been deprecated as of Python 2.6>
}}}
-Line 25:
+Line 66:
-== Launchpad source dependencies (sourcecode bzr trees) ==
+=== FunctionalLayer: OK (174 tests) ===
=== LaunchpadFunctionalLayer ===
  * valid_absolute_url (canonical.launchpad.validators.url) -- Python 2.6's urlparse disregards the uses_netloc list when parsing urls and treats anything beginning with // as netloc based. This makes maxb and wgrant say WTF!?! The untested valid_absolute_url DB constraint is also affected!
   * This behavior is [[http://bugs.python.org/issue7904#msg99265|expected]] and is in compliance with rfc3986, so we just need to change/remove our test. -- Salgado
-Line 27:
+Line 71:
-The `Makefile`s of pygettextpo & pygpgme make direct reference to Python 2.4, you'll need to amend them appropriately.
+=== AppServerLayer: OK (143 tests) ===
=== DatabaseFunctionalLayer ===
 * lib/lp/registry/tests/../doc/gpg-signatures.txt {{{
Failed doctest test for gpg-signatures.txt
  File "lib/lp/registry/tests/../doc/gpg-signatures.txt", line 0
-Line 29:
+Line 77:
-== Test Status for lp:~maxb/launchpad/python2.5@9026 based on lp:launchpad/stable@9539 ==
+----------------------------------------------------------------------
File "lib/lp/registry/tests/../doc/gpg-signatures.txt", line 138, in gpg-signatures.txt
Failed example:
    master_sig = gpghandler.getVerifiedSignature(content)
Differences (ndiff with -expected +actual):
      Traceback (most recent call last):
    - ...
    - GPGVerificationError...
    +   File "/home/salgado/devel/lp-sourcedeps/2.6/eggs/zope.testing-3.9.4-py2.6.egg/zope/testing/doctest/__init__.py", line 1355, in __run
    +     compileflags, 1) in test.globs
    +   File "<doctest gpg-signatures.txt[line 138, example 25]>", line 1, in <module>
    +   File "/home/salgado/devel/launchpad/python2.6/lib/canonical/launchpad/utilities/gpghandler.py", line 173, in getVerifiedSignature
    +     raise GPGVerificationError(e.message)
    + DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
}}}
-Line 31:
+Line 93:
- * ''canonical.testing.layers.!BaseLayer tests: Ran 256 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!LibrarianLayer tests: Ran 8 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!DatabaseLayer tests: Ran 43 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!LaunchpadLayer tests: Ran 23 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!ZopelessLayer tests: Ran 19 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!LaunchpadScriptLayer tests: Ran 72 tests with 0 failures and 0 errors''
 * canonical.testing.layers.!LaunchpadZopelessLayer tests: Ran 3192 tests with '''2 failures''' and 0 errors
  1. lib/lp/bugs/doc/checkwatches-cli-switches.txt: Looks to be formatting differences in help message
  1. lib/lp/bugs/doc/externalbugtracker-mantis-csv.txt: Sort order problem.
 * ''canonical.launchpad.ftests.test_system_documentation.!ProcessMailLayer tests: Ran 269 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!FunctionalLayer tests: Ran 164 tests with 0 failures and 0 errors''
 * canonical.testing.layers.!LaunchpadFunctionalLayer tests: Ran 8001 tests with '''11 failures''' and 0 errors
  1. lib/canonical/launchpad/ftests/../doc/pagetest-helpers.txt: CDATA section introduced
  1. lib/canonical/launchpad/ftests/../doc/tales.txt: Exception message concerning no len() changed in Py2.5.
  1. lib/canonical/launchpad/ftests/../doc/unicode_csv.txt: Py2.5 behaviour change in comparison of unicode / nonunicode
  1. lib/canonical/launchpad/scripts/ftests/librarianformatter.txt: Spurious characters.
  1. lib/lp/bugs/tests/externalbugtracker-xmlrpc-transport.txt: urllib2 seems to be behaving differently concerning installed handlers
  1. lib/lp/registry/browser/tests/person-views.txt: bytestring vs unicode issue
  1. lp.soyuz.scripts.tests.test_copypackage.!ReUploadFileTestCase.test_re_upload_file_does_not_leak_file_descriptors - `1 != 0`
  1. lib/lp/translations/tests/../doc/poexport-language-pack.txt: Hints at tarfile.getnames() behaviour change as seen in unit tests too
  1. lib/lp/translations/tests/../doc/poexport-request-productseries.txt: Ditto.
  1. lib/lp/translations/tests/../doc/poexport-request.txt: Ditto.
  1. lib/lp/translations/tests/../doc/potemplate.txt: Ditto.
 * ''canonical.testing.layers.!AppServerLayer tests: Ran 72 tests with 0 failures and 0 errors''
 * canonical.testing.layers.!DatabaseFunctionalLayer tests: Ran 3677 tests with '''1 failures''' and 0 errors
  1. lib/lp/registry/doc/teammembership.txt: Looks like benign formatting issue.
 * canonical.testing.layers.!PageTestLayer tests: Ran 5965 tests with '''7 failures''' and 0 errors
  1. lib/canonical/launchpad/pagetests/feeds/xx-links.txt: No longer replaces " with &amp;quot;.
  1. lib/lp/answers/stories/question-add.txt: Escaping issues as above, but with < and > too.
  1. lib/lp/answers/stories/question-obfuscation.txt: Ditto.
  1. lib/lp/bugs/stories/feeds/xx-bug-atom.txt: Looks like benign formatting issue
  1. lib/lp/bugs/stories/feeds/xx-bug-html.txt: Looks like benign formatting issue.
  1. lib/lp/bugs/stories/distribution/xx-distribution-upstream-bug-report.txt: Formatting issue.
  1. lib/lp/registry/tests/../stories/foaf/xx-person-working-on.txt: bytestring vs unicode issue
 * ''canonical.testing.layers.!TwistedLayer tests: Ran 219 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!TwistedLaunchpadZopelessLayer tests: Ran 23 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!TwistedAppServerLayer tests: Ran 5 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!ZopelessAppServerLayer tests: Ran 73 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!ZopelessDatabaseLayer tests: Ran 192 tests with 0 failures and 0 errors''
 * ''lp.codehosting.tests.test_acceptance.SSHServerLayer tests: Ran 46 tests with 0 failures and 0 errors''
 * zope.testing.testrunner.layer.!UnitTests tests: Ran 1378 tests with '''8 failures''' and 0 errors
  1. librarianformatter_noca.txt: Py2.5 differences in traceback formatting
  1. test_helpers.make_test_tarball_2: odd discrepancy
  1. test_raising_with_unprintable_exception (canonical.launchpad.webapp.tests.test_errorlog.!TestErrorReportingUtility): looks benign
  1. test_!EscapedQuotesPropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!InvalidPropertyFileUnicodeEscape (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!MultilinePropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!TrailingBackslashPropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!UnicodeEscapedPropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
+=== GoogleLaunchpadFunctionalLayer: OK (72 tests) ===
=== PageTestLayer: OK (6335 tests) ===
=== TwistedLayer: OK (235 tests) ===
=== TwistedLaunchpadZopelessLayer: OK (16 tests) ===
=== TwistedAppServerLayer: OK (9 tests) ===
=== ZopelessAppServerLayer: OK ===
=== ZopelessDatabaseLayer: OK (259 tests) ===
=== BugsWindmillLayer: OK (13 tests) ===
=== CodeWindmillLayer: OK (12 tests) ===
=== ForeignBranchPluginLayer: OK (61 tests) ===
=== SSHServerLayer: OK (46 tests) ===
=== RegistryWindmillLayer: OK (17 tests) ===
=== SoyuzWindmillLayer: OK (2 tests) ===
=== TestZopeTestInSubProcessLayer: OK (1 test) ===
=== TranslationsWindmillLayer: OK (7 tests) ===
=== UnitTests: OK ===
=== Shipit: OK === 
=== MailmanLayer: OK ===

launchpad development

Diff for "PythonMigrationStatus"

Moving to newer Python versions

Branches

Issues that need revisiting

Setup caveats

Known test failures and notes thereon

BaseLayer: OK (281 tests)

MemcachedLayer: OK (8 tests)

LibrarianLayer: OK (9 tests)

DatabaseLayer: OK (46 tests)

LaunchpadLayer: OK (24 tests)

ZopelessLayer: OK (20 tests)

LaunchpadScriptLayer: OK (85 tests)

LaunchpadZopelessLayer

FunctionalLayer: OK (174 tests)

LaunchpadFunctionalLayer

AppServerLayer: OK (143 tests)

DatabaseFunctionalLayer

GoogleLaunchpadFunctionalLayer: OK (72 tests)

PageTestLayer: OK (6335 tests)

TwistedLayer: OK (235 tests)

TwistedLaunchpadZopelessLayer: OK (16 tests)

TwistedAppServerLayer: OK (9 tests)

ZopelessAppServerLayer: OK

ZopelessDatabaseLayer: OK (259 tests)

BugsWindmillLayer: OK (13 tests)

CodeWindmillLayer: OK (12 tests)

ForeignBranchPluginLayer: OK (61 tests)

SSHServerLayer: OK (46 tests)

RegistryWindmillLayer: OK (17 tests)

SoyuzWindmillLayer: OK (2 tests)

TestZopeTestInSubProcessLayer: OK (1 test)

TranslationsWindmillLayer: OK (7 tests)

UnitTests: OK

MailmanLayer: OK