Diff for "PythonMigrationStatus"

Not logged in - Log In / Register
Differences between revisions 1 and 140 (spanning 139 versions)
⇤ ← Revision 1 as of 2009-10-09 17:47:27 →
Size: 6491
Editor: barry
Comment:
← Revision 140 as of 2010-04-12 22:49:20 → ⇥
Size: 26063
Editor: maxb
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
The ultimate goal is to migrate Launchpad to Python 2.6. Rather than go straight to Python 2.6, we are focusing on Python 2.5 first as a stepping stone. People involved in this effort: maxb, gary, salgado, barry

https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade

== Interesting branches ==

 * https://code.launchpad.net/~maxb/launchpad/python2.5 tracks `lp:launchpad/stable` (or occasionally `devel`), on a manual basis, and acts as a one-stop-shop for something that people can check out and build using Python 2.5

 * https://code.launchpad.net/~gary/launchpad/py25 is gary's branch tracking the upgrade.

 * ''add yours here''

== Launchpad code itself ==

Remaining changes in `python2.5` branch:
 * Changed `PYTHON_VERSION` in Makefiles
 * Changed shebang in `cronscripts/*`, `scripts/*`,

== Known issues ==

 * The pytz source package has a [[https://launchpad.net/bugs/438634|mysterious problem]]. The work around is to [[http://pypi.python.org/packages/2.5/p/pytz/pytz-2009l-py2.5.egg#md5=b41c8cacb5f4a8a64db05afbdf3ef9ef|download the 2.5 egg]] and put that in your `download-cache/dist` directory.

== Launchpad source dependencies (sourcecode bzr trees) ==

The `Makefile`s of pygettextpo & pygpgme make direct reference to Python 2.4, you'll need to amend them appropriately.

== Test Status for lp:~maxb/launchpad/python2.5@9026 based on lp:launchpad/stable@9539 ==

 * ''canonical.testing.layers.!BaseLayer tests: Ran 256 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!LibrarianLayer tests: Ran 8 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!DatabaseLayer tests: Ran 43 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!LaunchpadLayer tests: Ran 23 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!ZopelessLayer tests: Ran 19 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!LaunchpadScriptLayer tests: Ran 72 tests with 0 failures and 0 errors''
 * canonical.testing.layers.!LaunchpadZopelessLayer tests: Ran 3192 tests with '''2 failures''' and 0 errors
  1. lib/lp/bugs/doc/checkwatches-cli-switches.txt: Looks to be formatting differences in help message
  1. lib/lp/bugs/doc/externalbugtracker-mantis-csv.txt: Sort order problem.
 * ''canonical.launchpad.ftests.test_system_documentation.!ProcessMailLayer tests: Ran 269 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!FunctionalLayer tests: Ran 164 tests with 0 failures and 0 errors''
 * canonical.testing.layers.!LaunchpadFunctionalLayer tests: Ran 8001 tests with '''11 failures''' and 0 errors
  1. lib/canonical/launchpad/ftests/../doc/pagetest-helpers.txt: CDATA section introduced
  1. lib/canonical/launchpad/ftests/../doc/tales.txt: Exception message concerning no len() changed in Py2.5.
  1. lib/canonical/launchpad/ftests/../doc/unicode_csv.txt: Py2.5 behaviour change in comparison of unicode / nonunicode
  1. lib/canonical/launchpad/scripts/ftests/librarianformatter.txt: Spurious characters.
  1. lib/lp/bugs/tests/externalbugtracker-xmlrpc-transport.txt: urllib2 seems to be behaving differently concerning installed handlers
  1. lib/lp/registry/browser/tests/person-views.txt: bytestring vs unicode issue
  1. lp.soyuz.scripts.tests.test_copypackage.!ReUploadFileTestCase.test_re_upload_file_does_not_leak_file_descriptors - `1 != 0`
  1. lib/lp/translations/tests/../doc/poexport-language-pack.txt: Hints at tarfile.getnames() behaviour change as seen in unit tests too
  1. lib/lp/translations/tests/../doc/poexport-request-productseries.txt: Ditto.
  1. lib/lp/translations/tests/../doc/poexport-request.txt: Ditto.
  1. lib/lp/translations/tests/../doc/potemplate.txt: Ditto.
 * ''canonical.testing.layers.!AppServerLayer tests: Ran 72 tests with 0 failures and 0 errors''
 * canonical.testing.layers.!DatabaseFunctionalLayer tests: Ran 3677 tests with '''1 failures''' and 0 errors
  1. lib/lp/registry/doc/teammembership.txt: Looks like benign formatting issue.
 * canonical.testing.layers.!PageTestLayer tests: Ran 5965 tests with '''7 failures''' and 0 errors
  1. lib/canonical/launchpad/pagetests/feeds/xx-links.txt: No longer replaces " with ".
  1. lib/lp/answers/stories/question-add.txt: Escaping issues as above, but with < and > too.
  1. lib/lp/answers/stories/question-obfuscation.txt: Ditto.
  1. lib/lp/bugs/stories/feeds/xx-bug-atom.txt: Looks like benign formatting issue
  1. lib/lp/bugs/stories/feeds/xx-bug-html.txt: Looks like benign formatting issue.
  1. lib/lp/bugs/stories/distribution/xx-distribution-upstream-bug-report.txt: Formatting issue.
  1. lib/lp/registry/tests/../stories/foaf/xx-person-working-on.txt: bytestring vs unicode issue
 * ''canonical.testing.layers.!TwistedLayer tests: Ran 219 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!TwistedLaunchpadZopelessLayer tests: Ran 23 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!TwistedAppServerLayer tests: Ran 5 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!ZopelessAppServerLayer tests: Ran 73 tests with 0 failures and 0 errors''
 * ''canonical.testing.layers.!ZopelessDatabaseLayer tests: Ran 192 tests with 0 failures and 0 errors''
 * ''lp.codehosting.tests.test_acceptance.SSHServerLayer tests: Ran 46 tests with 0 failures and 0 errors''
 * zope.testing.testrunner.layer.!UnitTests tests: Ran 1378 tests with '''8 failures''' and 0 errors
  1. librarianformatter_noca.txt: Py2.5 differences in traceback formatting
  1. test_helpers.make_test_tarball_2: odd discrepancy
  1. test_raising_with_unprintable_exception (canonical.launchpad.webapp.tests.test_errorlog.!TestErrorReportingUtility): looks benign
  1. test_!EscapedQuotesPropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!InvalidPropertyFileUnicodeEscape (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!MultilinePropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!TrailingBackslashPropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)
  1. test_!UnicodeEscapedPropertyFileTest (lp.translations.utilities.tests.test_xpi_properties_format.!PropertyFileFormatTestCase)		 <<TableOfContents>>

With Lucid fast approaching, efforts to actually migrate Launchpad to Python 2.6 are getting under way.

Relevant tagged bugs: https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade

== Branches ==

 * https://code.launchpad.net/~launchpad-committers/launchpad/python2.6 -- collaboration branch created by maxb based on salgado's personal branch
 * https://code.launchpad.net/~salgado/shipit/shipit-python2.6

== Known/Open issues ==

 * Things partially dealt with by Salgado, who's going on leave and might not be able to finish them:
  * The failure when setting up the GoogleServiceLayer seems to be a consequence of the server answering with a errno.ECONNABORTED when it's not yet ready, so I've just changed the wait_for_service function to treat that error the same way it does for errno.ECONNREFUSED.
  * The test hang is because of the change done on bug Bug:349316, so I've just commented out that addCleanup() line for now, to see if we can get a full test run.

With the two changes above I was able to get the test suite to run to completion:
Total: 25856 tests, 71 failures, 119 errors in 300 minutes 55.792 seconds.

Here's the full log: [[attachment:tests.log]]

 * This appears to have been run without the Lucid pygpgme fix, so there are lots of "Unknown error code" failures in Soyuz and a couple of Registry tests. I'm rerunning the Soyuz-related tests to see what the real failures are. -- ''wgrant''

=== Things needing revising ===

A multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests would fail merely because of unexpected !DeprecationWarnings, so we've had to jump through multiple hoops all over the place to suppress these. Ultimately we need a plan for removing these suppressions, because we ''want'' to know about deprecation long-term.

=== Setup caveat ===
 * The extension modules in your `sourcecode` directory will have been built for Python 2.5 and these will break for your 2.6 branch. The simple thing to do is to move your old `sourcecode` aside and run `utilities/rocketfuel-get` to get it again with Python 2.6. A more complex but useful thing to do to manage `sourcecode` directories for multiple python versions is: {{{
cd ~/launchpad/lp-sourcedeps
mkdir 2.6
cd 2.6
ln -s ../download-cache
ln -s ../eggs
mkdir sourcecode
cd sourcecode
for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done
}}} and now point `utilities/link-external-sourcecode` to `../../lp-sourcedeps/2.6` - if you do that you'll need to be aware that `rocketfuel-get` won't know about it, and your extra directories will require occasional manual updating.

=== Cleanup potential ===
 * Is there any reason we need ipython-0.9.1-py2.5.egg in egg, not tarball, form?
  * Presumably we could bin the wadllib 0.1 eggs too.
 * Eventually we should get rid of all the scripts hardcoded `/usr/bin/pythonX.Y` and use buildout's shebang munging.
 * `scripts/branch-rewrite.py` uses `/usr/bin/pythonX.Y -u`. We should figure out if this is significant and why this script has to be different from other scripts.
 * `scripts/ftpmaster-tools/_syncorigins.py` uses `/usr/bin/env pythonX.Y`... why?!
 * `cronscripts/mirror-prober.sh` and `cronscripts/nightly.sh` are invoking python scripts with explicit interpreters. Why are they not using the shebangs? This leads to versions being defined in two places.

== Known test failures and notes thereon ==
=== BaseLayer: OK (281 tests) ===
=== MemcachedLayer: OK (8 tests) ===
=== LibrarianLayer: OK (9 tests) ===
=== DatabaseLayer: OK (46 tests) ===
=== LaunchpadLayer: OK (24 tests) ===
=== ZopelessLayer: OK (20 tests) ===

=== MailmanLayer: ?? ===
  * Note: These do not run automatically. Use `bin/test --layer=MailmanLayer`
  * You may get failures when you run all the tests in one go. Try running each individually for more accurate results.

=== LaunchpadScriptLayer ===
  * test_initZopelessTwice (canonical.lp.ftests.test_zopeless.TestInitZopeless) {{{
Error in test test_initZopelessTwice (canonical.lp.ftests.test_zopeless.TestInitZopeless)
Traceback (most recent call last):
  File "/usr/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/lp/ftests/test_zopeless.py", line 45, in test_initZopelessTwice
    dbuser='launchpad')
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/lp/__init__.py", line 84, in initZopeless
    dbname=dbname, dbhost=dbhost, dbuser=dbuser, isolation=isolation)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/database/sqlbase.py", line 336, in initZopeless
    warnings.warn(alreadyInstalledMsg, stacklevel=3)
UserWarning: A ZopelessTransactionManager with these settings is already installed. This is probably caused by calling initZopeless twice.
}}}

=== LaunchpadZopelessLayer ===
  * test_generateConfig (lp.archivepublisher.tests.test_ftparchive.TestFTPArchive): Is benign, caused by apt-ftparchive generating SHA-1 and SHA-256 checksums in more places in Lucid.
  * test_bug_496988 (lp.bugs.scripts.tests.test_checkwatches.TestCheckwatchesWithSyncableGnomeProducts) {{{
Traceback (most recent call last):
_StringException: Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
    return fn(*args)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
    testMethod()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/scripts/tests/test_checkwatches.py", line 109, in test_bug_496988
    gnome_bugzilla, [bug_watch_1, bug_watch_2])
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/scripts/checkwatches/updater.py", line 522, in _getExternalBugTrackersAndWatches
    remote_bug_ids))
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/bugzilla.py", line 617, in getProductsForRemoteBugs
    self.initializeRemoteBugDB(bug_ids)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/isolation.py", line 64, in wrapper
    return func(*args, **kwargs)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/bugzilla.py", line 568, in initializeRemoteBugDB
    'permissive': True,
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/xmlrpc.py", line 98, in request
    response = self._parse_response(self.opener.open(request), None)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault Client: "Content-Type must be 'text/xml,' 'multipart/*,' or 'application/dime' instead of 'application/x-www-form-urlencoded'">
------------
}}}
  * lib/lp/soyuz/tests/../doc/manage-chroot.txt is benign, Python 2.6 changed exception message format
  * lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt {{{
Failure in test lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt
Failed doctest test for bugs-emailinterface.txt
  File "lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt", line 2808, in bugs-emailinterface.txt
Failed example:
    print_attachments(get_latest_added_bug().attachments)
Differences (ndiff with -expected +actual):
    - LibraryFileAlias unnamed image/jpeg; name="image.jpg" UNSPECIFIED
    ? ^^^ ^ ^
    + LibraryFileAlias image.jpg image/jpeg; name="image.jpg" UNSPECIFIED
    ? ^^ ^ ^^^^
      this is not a real JPG file
      LibraryFileAlias sourcefile.diff text/x-diff; name="sourcefile1.diff" PATCH
      this should be diff output.
}}}
 * lib/canonical/launchpad/ftests/../doc/emailauthentication.txt {{{
Failure in test lib/canonical/launchpad/ftests/../doc/emailauthentication.txt
Failed doctest test for emailauthentication.txt
  File "lib/canonical/launchpad/ftests/../doc/emailauthentication.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/launchpad/ftests/../doc/emailauthentication.txt", line 150, in emailauthentication.txt
Failed example:
    print msg.as_string() #doctest: -NORMALIZE_WHITESPACE
Differences (ndiff with -expected +actual):
    - Date:...
    - ...
    + Date: Wed, 13 Apr 2005 19:18:42 -0400
    + From: Sample Person <test@canonical.com>
    + To: someone@somhost
    + Subject: Signed Email
    + Message-ID: <20050413231842.GB16989@localhost.localdomain>
    + Mime-Version: 1.0
    + Content-Type: multipart/signed; micalg=pgp-sha1;
    + protocol="application/pgp-signature"; boundary="8NvZYKFJsRX2Djef"
    + Content-Disposition: inline
    + Status: RO
    + <BLANKLINE>
    + <BLANKLINE>
    + --8NvZYKFJsRX2Djef
    + Content-Type: multipart/mixed;
    - Content-Type: multipart/mixed; boundary="--------------------EuxKj2iCbKjpUGkD"
    ? ------------------------------
    + boundary="--------------------EuxKj2iCbKjpUGkD"
    - ...
    + Content-Disposition: inline
    + <BLANKLINE>
    + <BLANKLINE>
    + ----------------------EuxKj2iCbKjpUGkD
    + Content-Type: text/plain; charset=us-ascii
    + Content-Disposition: inline
    + <BLANKLINE>
    + Some signed content.
    + <BLANKLINE>
    + ----------------------EuxKj2iCbKjpUGkD
    + Content-Type: text/plain; charset=us-ascii
    + Content-Disposition: attachment; filename="test.txt"
    + <BLANKLINE>
    + A signed attachment.
    + <BLANKLINE>
    + ----------------------EuxKj2iCbKjpUGkD--
    + <BLANKLINE>
    + --8NvZYKFJsRX2Djef
    + Content-Type: application/pgp-signature; name="signature.asc"
    + Content-Description: Digital signature
    + Content-Disposition: inline
    + -----BEGIN PGP SIGNATURE-----
    + Version: GnuPG v1.4.1 (GNU/Linux)
    + <BLANKLINE>
    + iD8DBQBDoAFKuiuTid/SBUMRAvzNAKCPHJgldGMFXh5SvKEufqB8N6IkoACdEu+8
    + Go2JWmGWaY8Nqquwk/Tr0pU=
    + =OhN7
    + -----END PGP SIGNATURE-----
    + <BLANKLINE>
    + --8NvZYKFJsRX2Djef--
    + <BLANKLINE>
    + <BLANKLINE>
    + <BLANKLINE>
    + <BLANKLINE>
}}}
  * lib/canonical/launchpad/ftests/../doc/incomingmail.txt is benign, Python 2.6 changed exception message format

=== FunctionalLayer: OK (174 tests) ===

=== LaunchpadFunctionalLayer ===
  * lib/canonical/launchpad/ftests/../doc/gpghandler.txt is benign, Python 2.6 changed exception message format
  * lib/canonical/launchpad/ftests/../doc/profiling.txt {{{
Failed doctest test for profiling.txt
  File "lib/canonical/launchpad/ftests/../doc/profiling.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/launchpad/ftests/../doc/profiling.txt", line 43, in profiling.txt
Failed example:
    http = HTTPCaller(host='launchpad.dev')
Exception raised:
    Traceback (most recent call last):
      File "/home/maxb/launchpad/lp-sourcedeps/eggs/zope.testing-3.8.1-py2.6.egg/zope/testing/doctest.py", line 1361, in __run
        compileflags, 1) in test.globs
      File "<doctest profiling.txt[line 43, example 12]>", line 1, in <module>
        http = HTTPCaller(host='launchpad.dev')
      File "/home/maxb/launchpad/lp-sourcedeps/eggs/zope.app.testing-3.7.3-py2.6.egg/zope/app/testing/functional.py", line 414, in __init__
        super(CookieHandler, self).__init__(*args, **kw)
    TypeError: object.__init__() takes no parameters
----------------------------------------------------------------------
}}}
  * lib/canonical/launchpad/scripts/ftests/librarianformatter.txt is being broken by !DeprecationWarnings (from cscvs and canonical.uuid)
  * valid_absolute_url (canonical.launchpad.validators.url) {{{
Failure in test valid_absolute_url (canonical.launchpad.validators.url)
Traceback (most recent call last):
  File "/usr/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/usr/lib/python2.6/doctest.py", line 2152, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for canonical.launchpad.validators.url.valid_absolute_url
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/launchpad/validators/url.py", line 11, in valid_absolute_url

----------------------------------------------------------------------
File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/launchpad/validators/url.py", line 24, in canonical.launchpad.validators.url.valid_absolute_url
Failed example:
    valid_absolute_url('whatever://example.com/blah')
Expected:
    False
Got:
    True
}}}
  * lib/lp/soyuz/tests/../doc/buildd-dispatching.txt is benign, Python 2.6 changed exception message format
  * lib/lp/soyuz/tests/../doc/buildd-sequencer.txt - !DeprecationWarning: the !MimeWriter module is deprecated; use the email package instead ([[http://twistedmatrix.com/trac/ticket/4038|Twisted bug #4038]], but [[Bug:344165|Soyuz bug #344165]] says that the script involved should be removed).
  * lib/lp/soyuz/tests/../doc/soyuz-upload.txt - Is benign, caused by apt-ftparchive generating SHA-1 and SHA-256 checksums in more places in Lucid.
  * lib/lp/translations/tests/../doc/poexport-language-pack.txt - looks awfully familiar from the 2.5 migration and is most likely due to atrocious compatibility of the tarfile module across versions, w.r.t. slashes on the end of directory names.
  * lib/lp/translations/tests/../doc/poexport-request-productseries.txt - ditto
  * lib/lp/translations/tests/../doc/poexport-request.txt - ditto
  * lib/lp/translations/tests/../doc/potemplate.txt - ditto
  * lib/lp/translations/utilities/tests/../doc/gettext_mo_exporter.txt - ditto
  * lib/lp/translations/utilities/tests/../doc/launchpad_write_tarfile.txt - ditto

=== AppServerLayer: OK (143 tests) ===
=== DatabaseFunctionalLayer: OK (4408 tests) ===

=== GoogleServiceLayer ===
Fails during layer setup. See salgado's comments earlier on this page. Entire layer does not run.

=== PageTestLayer ===
Depends on !GoogleServiceLayer so fails as per above. Entire layer does not run.

=== TwistedLayer: OK (235 tests) ===
=== TwistedLaunchpadZopelessLayer: OK (16 tests) ===

=== TwistedAppServerLayer ===
  * test_import_hg (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegration) {{{
Failure in test test_import_hg (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegration)
Traceback (most recent call last):
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 371, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 674, in assertImported
    self.assertCodeImportResultCreated(code_import)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 657, in assertCodeImportResultCreated
    self.assertEqual(result.status, CodeImportResultStatus.SUCCESS)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 287, in failUnlessEqual
    % (msg, pformat(first), pformat(second)))
FailTest: not equal:
a = <DBItem CodeImportResultStatus.FAILURE, (200) Failure>
b = <DBItem CodeImportResultStatus.SUCCESS, (100) Success>
}}}
  * test_import_hg (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegrationScript) - ditto
  * lp.codehosting.puller.tests.test_scheduler.TestPullerMasterIntegration.test_lock_with_magic_id - !DeprecationWarning from cscvs
  * test_mirror_with_destination_locked_by_another (lp.codehosting.puller.tests.test_scheduler.TestPullerMasterIntegration) - ditto

 * !ZopelessAppServerLayer:
  * lp.codehosting.puller.tests.test_acceptance.TestBranchPuller.test_mirror_hosted_loom_branch - TEST HUNG!
   '''Running just './bin/test -vv lp.codehosting.puller.tests.test_acceptance' seems to reproduce it, although it hangs at a different test every time.'''

TEST RUN INCOMPLETE - Following layers run manually individually

=== ZopelessDatabaseLayer ===
  * test_templates_tarball (canonical.buildd.tests.test_generate_translation_templates.TestGenerateTranslationTemplates) - I am somewhat suspicious that this might be the fault of the tarfile module too.

=== BugsWindmillLayer: OK (12 tests) ===
=== CodeWindmillLayer: OK (11 tests) ===


=== ForeignBranchPluginLayer ===
  * test_import (lp.codehosting.codeimport.tests.test_worker.TestMercurialImport)
 {{{
Error in test test_import (lp.codehosting.codeimport.tests.test_worker.TestMercurialImport)
Traceback (most recent call last):
_StringException: Text attachment: log
------------
39.627 bzr-hg: using Mercurial 1.3.1
39.628 creating repository in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/.
39.630 creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x3197490> in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/
39.637 trying to create missing lock '/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/checkout/dirstate'
39.637 opening working tree '/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree'
39.641 bzr-hg: using Mercurial 1.3.1
39.644 bzr-hg: using Mercurial 1.3.1
39.644 bzr-hg: using Mercurial 1.3.1
39.645 bzr-hg: using Mercurial 1.3.1
39.646 creating repository in file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/.
39.649 creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x3197490> in file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/
39.654 Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
39.654 fetch up to rev {None}
39.655 bzr-hg: using Mercurial 1.3.1
39.655 bzr-hg: using Mercurial 1.3.1
39.661 bzr-hg: using Mercurial 1.3.1
39.671 examining 3382fd51dc44:3382fd51dc44
39.671 found new changeset 3382fd51dc44
39.684 Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
39.684 fetch up to rev {hg-v1:3382fd51dc448abc95fcbd7d489d0d0ff7011db3}
39.693 Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
39.693 fetch up to rev {None}
------------
Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
    return fn(*args)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
   testMethod()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 696, in test_import
    worker.run()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 421, in run
    return self._doImport()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 555, in _doImport
    self.pushBazaarWorkingTree(bazaar_tree)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 649, in pushBazaarWorkingTree
    self.db_file, bazaar_tree.branch.repository._transport)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 273, in put
    local_file = source_transport.get(filename)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/bzr-2.1.0-py2.6-linux-x86_64.egg/bzrlib/transport/local.py", line 167, in get
    self._translate_error(e, path)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/bzr-2.1.0-py2.6-linux-x86_64.egg/bzrlib/transport/__init__.py", line 308, in _translate_error
    raise errors.NoSuchFile(path, extra=e)
NoSuchFile: No such file: u'/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/hg-v2.db': [Errno 2] No such file or directory: u'/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/hg-v2.db'
------------
}}}
  * test_import_script (lp.codehosting.codeimport.tests.test_worker.TestMercurialImport)
 - retval 1
  * test_script_exit_codes (lp.codehosting.codeimport.tests.test_worker.TestMercurialImport)
 - retval 1
  * test_sync (lp.codehosting.codeimport.tests.test_worker.TestMercurialImport)
 - like test_import

=== SSHServerLayer: OK (46 tests) ===
=== RegistryWindmillLayer: OK (17 tests) ===
=== SoyuzWindmillLayer: OK (2 tests) ===
=== TestZopeTestInSubProcessLayer: OK (1 test) ===
=== TranslationsWindmillLayer: OK (7 tests) ===

=== UnitTests ===
  * make_test_tarball_2 (canonical.launchpad.tests.test_helpers) -- iiiiiiiiit's tarfile again!
  * canonical.launchpad.webapp.url: lp:~maxb/launchpad/c-l-w-url-doctest-2.6
  * lib/canonical/lazr/tests/../doc/timeout.txt -- benign change in exception message string
  * lp.archiveuploader.tests.test_dscfile.TestDscFile: lp:~maxb/launchpad/test_dscfile-deprecation

  * test_script (lp.translations.scripts.tests.test_validate_translations_file.TestValidateTranslationsFile)
 -- AssertionError: 0 != 1 -- yes that really tells me what was wrong... not - urgh, there seems to be test isolation problems - this fails in a `bin/test -u` run, but passes if run alone.
  * lib/lp/translations/tests/../doc/remove-upstream-translations-script.txt -- Exploding w.r.t. "The wrong site.py is imported. Scripts should usually be started with Launchpad's bin/py, or with a Python invoked with the -S flag." - urgh, there seems to be test isolation problems - this fails in a `bin/test -u` run, but passes if run alone.

=== Shipit (74 tests, 2 failures, 1 errors) ===
  * lib/canonical/shipit/tests/../doc/shipit.txt
{{{
Failed doctest test for shipit.txt
File "lib/canonical/shipit/tests/../doc/shipit.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/shipit/tests/../doc/shipit.txt", line 445, in shipit.txt
Failed example:
    out, err
Differences (ndiff with -expected +actual):
    - ('', '')
    + ('', '/home/salgado/devel/launchpad/real-python-2.6/lib/cscvs/dircompare/ dirsums.py:18: DeprecationWarning: the md5 module is deprecated; use hashlib instead\n import md5\n/home/salgado/devel/launchpad/real-python-2.6/lib/ canonical/uuid.py:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead\n import time, random, subprocess, thread, sha\n')
----------------------------------------------------------------------
File "lib/canonical/shipit/tests/../doc/shipit.txt", line 1211, in shipit.txt
Failed example:
    out, err
Differences (ndiff with -expected +actual):
    - ('', '')
    + ('', '/home/salgado/devel/launchpad/real-python-2.6/lib/canonical/uuid.py:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead\n import time, random, subprocess, thread, sha\n/home/salgado/devel/ launchpad/real-python-2.6/lib/cscvs/dircompare/dirsums.py:18: DeprecationWarning: the md5 module is deprecated; use hashlib instead\n import md5\n')
}}}
  * lib/canonical/shipit/tests/../doc/shipit-process-requests.txt
{{{
Failure in test lib/canonical/shipit/tests/../doc/shipit-process-requests.txt
Failed doctest test for shipit-process-requests.txt
  File "lib/canonical/shipit/tests/../doc/shipit-process-requests.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/shipit/tests/../doc/shipit-process-requests.txt", line 96, in shipit-process-requests.txt
Failed example:
    out, err
Differences (ndiff with -expected +actual):
    - ('', '')
    + ('', '/home/salgado/devel/launchpad/real-python-2.6/lib/cscvs/dircompare/ dirsums.py:18: DeprecationWarning: the md5 module is deprecated; use hashlib instead\n import md5\n/home/salgado/devel/launchpad/real-python-2.6/lib/ canonical/uuid.py:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead\n import time, random, subprocess, thread, sha\n')
}}}

Moving to newer Python versions

Contents

  1. Moving to newer Python versions
    1. Branches
    2. Known/Open issues
      1. Things needing revising
      2. Setup caveat
      3. Cleanup potential
    3. Known test failures and notes thereon
      1. BaseLayer: OK (281 tests)
      2. MemcachedLayer: OK (8 tests)
      3. LibrarianLayer: OK (9 tests)
      4. DatabaseLayer: OK (46 tests)
      5. LaunchpadLayer: OK (24 tests)
      6. ZopelessLayer: OK (20 tests)
      7. MailmanLayer: ??
      8. LaunchpadScriptLayer
      9. LaunchpadZopelessLayer
      10. FunctionalLayer: OK (174 tests)
      11. LaunchpadFunctionalLayer
      12. AppServerLayer: OK (143 tests)
      13. DatabaseFunctionalLayer: OK (4408 tests)
      14. GoogleServiceLayer
      15. PageTestLayer
      16. TwistedLayer: OK (235 tests)
      17. TwistedLaunchpadZopelessLayer: OK (16 tests)
      18. TwistedAppServerLayer
      19. ZopelessDatabaseLayer
      20. BugsWindmillLayer: OK (12 tests)
      21. CodeWindmillLayer: OK (11 tests)
      22. ForeignBranchPluginLayer
      23. SSHServerLayer: OK (46 tests)
      24. RegistryWindmillLayer: OK (17 tests)
      25. SoyuzWindmillLayer: OK (2 tests)
      26. TestZopeTestInSubProcessLayer: OK (1 test)
      27. TranslationsWindmillLayer: OK (7 tests)
      28. UnitTests
      29. Shipit (74 tests, 2 failures, 1 errors)

With Lucid fast approaching, efforts to actually migrate Launchpad to Python 2.6 are getting under way.

Relevant tagged bugs: https://bugs.launchpad.net/launchpad-foundations/+bugs?field.tag=python-upgrade

Branches

Known/Open issues

  • Things partially dealt with by Salgado, who's going on leave and might not be able to finish them:

    • The failure when setting up the GoogleServiceLayer seems to be a consequence of the server answering with a errno.ECONNABORTED when it's not yet ready, so I've just changed the wait_for_service function to treat that error the same way it does for errno.ECONNREFUSED.

    • The test hang is because of the change done on bug 349316, so I've just commented out that addCleanup() line for now, to see if we can get a full test run.

With the two changes above I was able to get the test suite to run to completion: Total: 25856 tests, 71 failures, 119 errors in 300 minutes 55.792 seconds.

Here's the full log: tests.log

  • This appears to have been run without the Lucid pygpgme fix, so there are lots of "Unknown error code" failures in Soyuz and a couple of Registry tests. I'm rerunning the Soyuz-related tests to see what the real failures are. -- wgrant

Things needing revising

A multitude of modules of modules (sha, md5, sets, ...) are deprecated in Python 2.6, and many tests would fail merely because of unexpected DeprecationWarnings, so we've had to jump through multiple hoops all over the place to suppress these. Ultimately we need a plan for removing these suppressions, because we want to know about deprecation long-term.

Setup caveat

  • The extension modules in your sourcecode directory will have been built for Python 2.5 and these will break for your 2.6 branch. The simple thing to do is to move your old sourcecode aside and run utilities/rocketfuel-get to get it again with Python 2.6. A more complex but useful thing to do to manage sourcecode directories for multiple python versions is: 

    cd ~/launchpad/lp-sourcedeps
mkdir 2.6
cd 2.6
ln -s ../download-cache
ln -s ../eggs
mkdir sourcecode
cd sourcecode
for i in ../../sourcecode/*; do bzr checkout --lightweight "$i"; done

    and now point utilities/link-external-sourcecode to ../../lp-sourcedeps/2.6 - if you do that you'll need to be aware that rocketfuel-get won't know about it, and your extra directories will require occasional manual updating.

Cleanup potential

  • Is there any reason we need ipython-0.9.1-py2.5.egg in egg, not tarball, form?
    • Presumably we could bin the wadllib 0.1 eggs too.

  • Eventually we should get rid of all the scripts hardcoded /usr/bin/pythonX.Y and use buildout's shebang munging.

  • scripts/branch-rewrite.py uses /usr/bin/pythonX.Y -u. We should figure out if this is significant and why this script has to be different from other scripts.

  • scripts/ftpmaster-tools/_syncorigins.py uses /usr/bin/env pythonX.Y... why?!

  • cronscripts/mirror-prober.sh and cronscripts/nightly.sh are invoking python scripts with explicit interpreters. Why are they not using the shebangs? This leads to versions being defined in two places.

Known test failures and notes thereon

BaseLayer: OK (281 tests)

MemcachedLayer: OK (8 tests)

LibrarianLayer: OK (9 tests)

DatabaseLayer: OK (46 tests)

LaunchpadLayer: OK (24 tests)

ZopelessLayer: OK (20 tests)

MailmanLayer: ??

  • Note: These do not run automatically. Use bin/test --layer=MailmanLayer

  • You may get failures when you run all the tests in one go. Try running each individually for more accurate results.

LaunchpadScriptLayer

  • test_initZopelessTwice (canonical.lp.ftests.test_zopeless.TestInitZopeless) 

    Error in test test_initZopelessTwice (canonical.lp.ftests.test_zopeless.TestInitZopeless)
Traceback (most recent call last):
  File "/usr/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/lp/ftests/test_zopeless.py", line 45, in test_initZopelessTwice
    dbuser='launchpad')
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/lp/__init__.py", line 84, in initZopeless
    dbname=dbname, dbhost=dbhost, dbuser=dbuser, isolation=isolation)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/database/sqlbase.py", line 336, in initZopeless
    warnings.warn(alreadyInstalledMsg, stacklevel=3)
UserWarning: A ZopelessTransactionManager with these settings is already installed.  This is probably caused by calling initZopeless twice.

LaunchpadZopelessLayer

  • test_generateConfig (lp.archivepublisher.tests.test_ftparchive.TestFTPArchive): Is benign, caused by apt-ftparchive generating SHA-1 and SHA-256 checksums in more places in Lucid.

  • test_bug_496988 (lp.bugs.scripts.tests.test_checkwatches.TestCheckwatchesWithSyncableGnomeProducts) 

    Traceback (most recent call last):
_StringException: Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
    return fn(*args)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
    testMethod()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/scripts/tests/test_checkwatches.py", line 109, in test_bug_496988
    gnome_bugzilla, [bug_watch_1, bug_watch_2])
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/scripts/checkwatches/updater.py", line 522, in _getExternalBugTrackersAndWatches
    remote_bug_ids))
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/bugzilla.py", line 617, in getProductsForRemoteBugs
    self.initializeRemoteBugDB(bug_ids)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/isolation.py", line 64, in wrapper
    return func(*args, **kwargs)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/bugzilla.py", line 568, in initializeRemoteBugDB
    'permissive': True,
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/bugs/externalbugtracker/xmlrpc.py", line 98, in request
    response = self._parse_response(self.opener.open(request), None)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
    return u.close()
  File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
Fault: <Fault Client: "Content-Type must be 'text/xml,' 'multipart/*,' or 'application/dime' instead of 'application/x-www-form-urlencoded'">
------------
  • lib/lp/soyuz/tests/../doc/manage-chroot.txt is benign, Python 2.6 changed exception message format

  • lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt 

    Failure in test lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt
Failed doctest test for bugs-emailinterface.txt
  File "lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/launchpad/ftests/../../../lp/bugs/tests/bugs-emailinterface.txt", line 2808, in bugs-emailinterface.txt
Failed example:
    print_attachments(get_latest_added_bug().attachments)
Differences (ndiff with -expected +actual):
    - LibraryFileAlias unnamed image/jpeg; name="image.jpg" UNSPECIFIED
    ?                  ^^^ ^ ^
    + LibraryFileAlias image.jpg image/jpeg; name="image.jpg" UNSPECIFIED
    ?                  ^^ ^ ^^^^
      this is not a real JPG file
      LibraryFileAlias sourcefile.diff text/x-diff; name="sourcefile1.diff" PATCH
      this should be diff output.

  • lib/canonical/launchpad/ftests/../doc/emailauthentication.txt 

    Failure in test lib/canonical/launchpad/ftests/../doc/emailauthentication.txt
Failed doctest test for emailauthentication.txt
  File "lib/canonical/launchpad/ftests/../doc/emailauthentication.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/launchpad/ftests/../doc/emailauthentication.txt", line 150, in emailauthentication.txt
Failed example:
    print msg.as_string() #doctest: -NORMALIZE_WHITESPACE
Differences (ndiff with -expected +actual):
    - Date:...
    - ...
    + Date: Wed, 13 Apr 2005 19:18:42 -0400
    + From: Sample Person <test@canonical.com>
    + To: someone@somhost
    + Subject: Signed Email
    + Message-ID: <20050413231842.GB16989@localhost.localdomain>
    + Mime-Version: 1.0
    + Content-Type: multipart/signed; micalg=pgp-sha1;
    +   protocol="application/pgp-signature"; boundary="8NvZYKFJsRX2Djef"
    + Content-Disposition: inline
    + Status: RO
    + <BLANKLINE>
    + <BLANKLINE>
    + --8NvZYKFJsRX2Djef
    + Content-Type: multipart/mixed;
    - Content-Type: multipart/mixed; boundary="--------------------EuxKj2iCbKjpUGkD"
    ? ------------------------------
    +  boundary="--------------------EuxKj2iCbKjpUGkD"
    - ...
    + Content-Disposition: inline
    + <BLANKLINE>
    + <BLANKLINE>
    + ----------------------EuxKj2iCbKjpUGkD
    + Content-Type: text/plain; charset=us-ascii
    + Content-Disposition: inline
    + <BLANKLINE>
    + Some signed content.
    + <BLANKLINE>
    + ----------------------EuxKj2iCbKjpUGkD
    + Content-Type: text/plain; charset=us-ascii
    + Content-Disposition: attachment; filename="test.txt"
    + <BLANKLINE>
    + A signed attachment.
    + <BLANKLINE>
    + ----------------------EuxKj2iCbKjpUGkD--
    + <BLANKLINE>
    + --8NvZYKFJsRX2Djef
    + Content-Type: application/pgp-signature; name="signature.asc"
    + Content-Description: Digital signature
    + Content-Disposition: inline
    + -----BEGIN PGP SIGNATURE-----
    + Version: GnuPG v1.4.1 (GNU/Linux)
    + <BLANKLINE>
    + iD8DBQBDoAFKuiuTid/SBUMRAvzNAKCPHJgldGMFXh5SvKEufqB8N6IkoACdEu+8
    + Go2JWmGWaY8Nqquwk/Tr0pU=
    + =OhN7
    + -----END PGP SIGNATURE-----
    + <BLANKLINE>
    + --8NvZYKFJsRX2Djef--
    + <BLANKLINE>
    + <BLANKLINE>
    + <BLANKLINE>
    + <BLANKLINE>
    • lib/canonical/launchpad/ftests/../doc/incomingmail.txt is benign, Python 2.6 changed exception message format

FunctionalLayer: OK (174 tests)

LaunchpadFunctionalLayer

  • lib/canonical/launchpad/ftests/../doc/gpghandler.txt is benign, Python 2.6 changed exception message format

  • lib/canonical/launchpad/ftests/../doc/profiling.txt 

    Failed doctest test for profiling.txt
  File "lib/canonical/launchpad/ftests/../doc/profiling.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/launchpad/ftests/../doc/profiling.txt", line 43, in profiling.txt
Failed example:
    http = HTTPCaller(host='launchpad.dev')
Exception raised:
    Traceback (most recent call last):
      File "/home/maxb/launchpad/lp-sourcedeps/eggs/zope.testing-3.8.1-py2.6.egg/zope/testing/doctest.py", line 1361, in __run
        compileflags, 1) in test.globs
      File "<doctest profiling.txt[line 43, example 12]>", line 1, in <module>
        http = HTTPCaller(host='launchpad.dev')
      File "/home/maxb/launchpad/lp-sourcedeps/eggs/zope.app.testing-3.7.3-py2.6.egg/zope/app/testing/functional.py", line 414, in __init__
        super(CookieHandler, self).__init__(*args, **kw)
    TypeError: object.__init__() takes no parameters
----------------------------------------------------------------------

  • lib/canonical/launchpad/scripts/ftests/librarianformatter.txt is being broken by DeprecationWarnings (from cscvs and canonical.uuid)

  • valid_absolute_url (canonical.launchpad.validators.url) 

    Failure in test valid_absolute_url (canonical.launchpad.validators.url)
Traceback (most recent call last):
  File "/usr/lib/python2.6/unittest.py", line 279, in run
    testMethod()
  File "/usr/lib/python2.6/doctest.py", line 2152, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for canonical.launchpad.validators.url.valid_absolute_url
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/launchpad/validators/url.py", line 11, in valid_absolute_url

----------------------------------------------------------------------
File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/canonical/launchpad/validators/url.py", line 24, in canonical.launchpad.validators.url.valid_absolute_url
Failed example:
    valid_absolute_url('whatever://example.com/blah')
Expected:
    False
Got:
    True
  • lib/lp/soyuz/tests/../doc/buildd-dispatching.txt is benign, Python 2.6 changed exception message format

  • lib/lp/soyuz/tests/../doc/buildd-sequencer.txt - DeprecationWarning: the MimeWriter module is deprecated; use the email package instead (Twisted bug #4038, but Soyuz bug #344165 says that the script involved should be removed).

  • lib/lp/soyuz/tests/../doc/soyuz-upload.txt - Is benign, caused by apt-ftparchive generating SHA-1 and SHA-256 checksums in more places in Lucid.
  • lib/lp/translations/tests/../doc/poexport-language-pack.txt - looks awfully familiar from the 2.5 migration and is most likely due to atrocious compatibility of the tarfile module across versions, w.r.t. slashes on the end of directory names.
  • lib/lp/translations/tests/../doc/poexport-request-productseries.txt - ditto
  • lib/lp/translations/tests/../doc/poexport-request.txt - ditto
  • lib/lp/translations/tests/../doc/potemplate.txt - ditto
  • lib/lp/translations/utilities/tests/../doc/gettext_mo_exporter.txt - ditto
  • lib/lp/translations/utilities/tests/../doc/launchpad_write_tarfile.txt - ditto

AppServerLayer: OK (143 tests)

DatabaseFunctionalLayer: OK (4408 tests)

GoogleServiceLayer

Fails during layer setup. See salgado's comments earlier on this page. Entire layer does not run.

PageTestLayer

Depends on GoogleServiceLayer so fails as per above. Entire layer does not run.

TwistedLayer: OK (235 tests)

TwistedLaunchpadZopelessLayer: OK (16 tests)

TwistedAppServerLayer

  • test_import_hg (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegration) 

    Failure in test test_import_hg (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegration)
Traceback (most recent call last):
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 371, in _runCallbacks
    self.result = callback(self.result, *args, **kw)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 674, in assertImported
    self.assertCodeImportResultCreated(code_import)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/tests/test_workermonitor.py", line 657, in assertCodeImportResultCreated
    self.assertEqual(result.status, CodeImportResultStatus.SUCCESS)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/Twisted-10.0.0-py2.6-linux-x86_64.egg/twisted/trial/unittest.py", line 287, in failUnlessEqual
    % (msg, pformat(first), pformat(second)))
FailTest: not equal:
a = <DBItem CodeImportResultStatus.FAILURE, (200) Failure>
b = <DBItem CodeImportResultStatus.SUCCESS, (100) Success>

  • test_import_hg (lp.codehosting.codeimport.tests.test_workermonitor.TestWorkerMonitorIntegrationScript) - ditto

  • lp.codehosting.puller.tests.test_scheduler.TestPullerMasterIntegration.test_lock_with_magic_id - DeprecationWarning from cscvs

  • test_mirror_with_destination_locked_by_another (lp.codehosting.puller.tests.test_scheduler.TestPullerMasterIntegration) - ditto

  • ZopelessAppServerLayer:

    • lp.codehosting.puller.tests.test_acceptance.TestBranchPuller.test_mirror_hosted_loom_branch - TEST HUNG!

      • Running just './bin/test -vv lp.codehosting.puller.tests.test_acceptance' seems to reproduce it, although it hangs at a different test every time.

TEST RUN INCOMPLETE - Following layers run manually individually

ZopelessDatabaseLayer

  • test_templates_tarball (canonical.buildd.tests.test_generate_translation_templates.TestGenerateTranslationTemplates) - I am somewhat suspicious that this might be the fault of the tarfile module too.

BugsWindmillLayer: OK (12 tests)

CodeWindmillLayer: OK (11 tests)

ForeignBranchPluginLayer

  • Error in test test_import (lp.codehosting.codeimport.tests.test_worker.TestMercurialImport)
Traceback (most recent call last):
_StringException: Text attachment: log
------------
39.627  bzr-hg: using Mercurial 1.3.1
39.628  creating repository in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/.
39.630  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x3197490> in file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/
39.637  trying to create missing lock '/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/checkout/dirstate'
39.637  opening working tree '/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree'
39.641  bzr-hg: using Mercurial 1.3.1
39.644  bzr-hg: using Mercurial 1.3.1
39.644  bzr-hg: using Mercurial 1.3.1
39.645  bzr-hg: using Mercurial 1.3.1
39.646  creating repository in file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/.
39.649  creating branch <bzrlib.branch.BzrBranchFormat7 object at 0x3197490> in file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/
39.654  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
39.654  fetch up to rev {None}
39.655  bzr-hg: using Mercurial 1.3.1
39.655  bzr-hg: using Mercurial 1.3.1
39.661  bzr-hg: using Mercurial 1.3.1
39.671  examining 3382fd51dc44:3382fd51dc44
39.671  found new changeset 3382fd51dc44
39.684  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
39.684  fetch up to rev {hg-v1:3382fd51dc448abc95fcbd7d489d0d0ff7011db3}
39.693  Using fetch logic to copy between CHKInventoryRepository('file:///var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/')(<RepositoryFormat2a>) and CHKInventoryRepository('file:///tmp/testbzr-SAHgvG.tmp/lp.codehosting.codeimport.tests.test_worker.TestMercurialImport.test_import/work/bazaar_store/00000001/.bzr/repository/')(<RepositoryFormat2a>)
39.693  fetch up to rev {None}
------------
Text attachment: traceback
------------
Traceback (most recent call last):
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/runtest.py", line 128, in _run_user
    return fn(*args)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/testtools-0.9.2-py2.6.egg/testtools/testcase.py", line 368, in _run_test_method
   testMethod()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/tests/test_worker.py", line 696, in test_import
    worker.run()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 421, in run
    return self._doImport()
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 555, in _doImport
    self.pushBazaarWorkingTree(bazaar_tree)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 649, in pushBazaarWorkingTree
    self.db_file, bazaar_tree.branch.repository._transport)
  File "/home/maxb/launchpad/lp-branches/real-python-2.6/lib/lp/codehosting/codeimport/worker.py", line 273, in put
    local_file = source_transport.get(filename)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/bzr-2.1.0-py2.6-linux-x86_64.egg/bzrlib/transport/local.py", line 167, in get
    self._translate_error(e, path)
  File "/home/maxb/launchpad/lp-sourcedeps/eggs/bzr-2.1.0-py2.6-linux-x86_64.egg/bzrlib/transport/__init__.py", line 308, in _translate_error
    raise errors.NoSuchFile(path, extra=e)
NoSuchFile: No such file: u'/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/hg-v2.db': [Errno 2] No such file or directory: u'/var/tmp/codeimport/data/worker-for-branch-1/bzr_working_tree/.bzr/repository/hg-v2.db'
------------
    - retval 1 - retval 1 - like test_import

SSHServerLayer: OK (46 tests)

RegistryWindmillLayer: OK (17 tests)

SoyuzWindmillLayer: OK (2 tests)

TestZopeTestInSubProcessLayer: OK (1 test)

TranslationsWindmillLayer: OK (7 tests)

UnitTests

  • make_test_tarball_2 (canonical.launchpad.tests.test_helpers) -- iiiiiiiiit's tarfile again!
  • canonical.launchpad.webapp.url: lp:~maxb/launchpad/c-l-w-url-doctest-2.6
  • lib/canonical/lazr/tests/../doc/timeout.txt -- benign change in exception message string

  • lp.archiveuploader.tests.test_dscfile.TestDscFile: lp:~maxb/launchpad/test_dscfile-deprecation

  • test_script (lp.translations.scripts.tests.test_validate_translations_file.TestValidateTranslationsFile)

  • -- AssertionError: 0 != 1 -- yes that really tells me what was wrong... not - urgh, there seems to be test isolation problems - this fails in a bin/test -u run, but passes if run alone.

    • lib/lp/translations/tests/../doc/remove-upstream-translations-script.txt -- Exploding w.r.t. "The wrong site.py is imported. Scripts should usually be started with Launchpad's bin/py, or with a Python invoked with the -S flag." - urgh, there seems to be test isolation problems - this fails in a bin/test -u run, but passes if run alone.

Shipit (74 tests, 2 failures, 1 errors)

  • lib/canonical/shipit/tests/../doc/shipit.txt 

Failed doctest test for shipit.txt
File "lib/canonical/shipit/tests/../doc/shipit.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/shipit/tests/../doc/shipit.txt", line 445, in shipit.txt
Failed example:
    out, err
Differences (ndiff with -expected +actual):
    - ('', '')
    + ('', '/home/salgado/devel/launchpad/real-python-2.6/lib/cscvs/dircompare/ dirsums.py:18: DeprecationWarning: the md5 module is deprecated; use hashlib    instead\n  import md5\n/home/salgado/devel/launchpad/real-python-2.6/lib/       canonical/uuid.py:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead\n  import time, random, subprocess, thread, sha\n')
----------------------------------------------------------------------
File "lib/canonical/shipit/tests/../doc/shipit.txt", line 1211, in shipit.txt
Failed example:
    out, err
Differences (ndiff with -expected +actual):
    - ('', '')
    + ('', '/home/salgado/devel/launchpad/real-python-2.6/lib/canonical/uuid.py:10: DeprecationWarning: the sha module is deprecated; use the hashlib module    instead\n  import time, random, subprocess, thread, sha\n/home/salgado/devel/   launchpad/real-python-2.6/lib/cscvs/dircompare/dirsums.py:18:                   DeprecationWarning: the md5 module is deprecated; use hashlib instead\n  import md5\n')
  • lib/canonical/shipit/tests/../doc/shipit-process-requests.txt 

Failure in test lib/canonical/shipit/tests/../doc/shipit-process-requests.txt
Failed doctest test for shipit-process-requests.txt
  File "lib/canonical/shipit/tests/../doc/shipit-process-requests.txt", line 0

----------------------------------------------------------------------
File "lib/canonical/shipit/tests/../doc/shipit-process-requests.txt", line 96,  in shipit-process-requests.txt
Failed example:
    out, err
Differences (ndiff with -expected +actual):
    - ('', '')
    + ('', '/home/salgado/devel/launchpad/real-python-2.6/lib/cscvs/dircompare/ dirsums.py:18: DeprecationWarning: the md5 module is deprecated; use hashlib    instead\n  import md5\n/home/salgado/devel/launchpad/real-python-2.6/lib/       canonical/uuid.py:10: DeprecationWarning: the sha module is deprecated; use the hashlib module instead\n  import time, random, subprocess, thread, sha\n')