Diff for "Hacking"

Not logged in - Log In / Register

Differences between revisions 1 and 30 (spanning 29 versions)
Revision 1 as of 2009-07-29 15:54:44
Size: 34810
Editor: gary
Comment:
Revision 30 as of 2021-08-09 16:45:23
Size: 30220
Editor: cjwatson
Comment: AFAIK there's no ordering randomization any more
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
See also: PythonStyleGuide. See also: PythonStyleGuide and HackingLazrLibraries.

Want to navigate the source tree? Look at our [[apidoc|API documentation]].
Line 14: Line 16:
Currently, Launchpad and Bazaar require Python 2.4. You may use features that require Python 2.4, as long as you abide by the PythonStyleGuide. Currently, Launchpad requires Python 3.5. You may use features that require Python 3.5, as long as you abide by the PythonStyleGuide.
Line 18: Line 20:
 First of all, thank you for writing docstrings. They make the code much easier to follow for the people that didn't originally write it. To answer the question, you have the following options. First of all, thank you for writing docstrings. They make the code much easier to follow for the people that didn't originally write it. To answer the question, you have the following options.
Line 36: Line 38:
 See ExceptionGuidelines. See ExceptionGuidelines.
Line 40: Line 42:
 In short, never raise {{{ValueError}}}, {{{NameError}}} or {{{TypeError}}}, and avoid subclassing these exceptions as well. The full instructions are at ExceptionGuidelines.

 In the case of {{{NotFoundError}}}, if you are going to catch this specific error in some other code, and then take some corrective action or some logging action, then seriously consider creating your own subclass. This allows your code to handle exactly the situation that you expect, and not be tricked into handling {{{NotFoundErrors}}} raised by code several levels in.

 When writing docstrings, always think whether it makes things clearer to say which exceptions will be raised due to inappropriate values being passed in.
In short, never raise {{{ValueError}}}, {{{NameError}}} or {{{TypeError}}}, and avoid subclassing these exceptions as well. The full instructions are at ExceptionGuidelines.

In the case of {{{NotFoundError}}}, if you are going to catch this specific error in some other code, and then take some corrective action or some logging action, then seriously consider creating your own subclass. This allows your code to handle exactly the situation that you expect, and not be tricked into handling {{{NotFoundErrors}}} raised by code several levels in.

When writing docstrings, always think whether it makes things clearer to say which exceptions will be raised due to inappropriate values being passed in.
Line 48: Line 50:
 For now, all self-posting forms have to call canonical.database.sqlbase.flush_database_updates() after processing the form. For now, all self-posting forms have to call lp.services.database.sqlbase.flush_database_updates() after processing the form.
Line 54: Line 56:
 Use an EnumCol. See the EnumCol wiki page for how it works. Use an EnumCol. See the EnumCol wiki page for how it works.
Line 58: Line 60:
 When you get a list from a security proxied object, that list is protected against being altered. This is important, because you don't know who else might have a reference to that list.

 When programming in Python generally, it is a good idea to make a copy of a list you get from somewhere before altering it. The security proxies just enforce this good practice.

 You can make a copy of the list by using the 'list' constructor. Here is an example, using the launchpad API.
 {{{
When you get a list from a security proxied object, that list is protected against being altered. This is important, because you don't know who else might have a reference to that list.

When programming in Python generally, it is a good idea to make a copy of a list you get from somewhere before altering it. The security proxies just enforce this good practice.

You can make a copy of the list by using the 'list' constructor. Here is an example, using the launchpad API.
{{{
Line 70: Line 72:
 You can also use the {{{sorted}}} builtin to do this.
 {{{
You can also use the {{{sorted}}} builtin to do this.
{{{
Line 85: Line 87:
For code that uses SQLObject, result sets will be randomised while conforming to the specified ORDER BY clause. This makes it difficult to write tests that depend on the natural ordering.

In contrast, code that does raw SQL won't have a randomised result set, so the natural order gets exposed to the user. The same rules of thumb apply when testing such code, but extra care should be taken not to rely on natural ordering.

Line 92: Line 89:
 You have to wrap it in an SQLConstant:

 {{{
You have to wrap it in an SQLConstant:

{{{
Line 100: Line 97:
 }}} }}}
Line 104: Line 101:
 The safest way is to ensure that all data is passed in as parameters, the way the DB-API intended it:

 {{{
 #!python
The safest way is to ensure that all data is passed in as parameters, the way the DB-API intended it:

{{{
#!python
Line 111: Line 108:
 }}}

 If you need to embed your data in the SQL query itself, there is only one rule you need to remember - quote your data. Failing to do this opens up the system to an SQL Injection attack, one of the more common and widely known security holes and also one of the more destructive. Don't attempt to write your own quote method - you will probably get it wrong and end up with the DBA removing precious parts of your anatomy. The only two formats you can use are %d and %s, and %s should always be escaped, *no exceptions!*

 {{{
 from canonical.database.sqlbase import quote
}}}

If you need to embed your data in the SQL query itself, there is only one rule you need to remember - quote your data. Failing to do this opens up the system to an SQL Injection attack, one of the more common and widely known security holes and also one of the more destructive. Don't attempt to write your own quote method - you will probably get it wrong and end up with the DBA removing precious parts of your anatomy. The only two formats you can use are %d and %s, and %s should always be escaped, *no exceptions!*

{{{
#!python
 from lp
.services.database.sqlbase import quote
Line 118: Line 116:
 cur.execute("select id,name from Person where displayname=%s" % quote("Stuart Bishop")  cur.execute("select id,name from Person where displayname=%s" % quote("Stuart Bishop"))
Line 122: Line 120:
   ''' '; drop table person cascade; insert into person(name) values ('hahaloser')''')
 }}}

 (the second command in the previous example demonstrates a simple argument that might be passed in by an attacker).
   "'; drop table person cascade; insert into person(name) values ('hahaloser')"))
}}}

(the second command in the previous example demonstrates a simple argument that might be passed in by an attacker).
Line 129: Line 127:
 See DatetimeUsageGuide for information on what types to use, and how the Python {{{datetime}}} types relate to database types through SQLObject. See DatetimeUsageGuide for information on what types to use, and how the Python {{{datetime}}} types relate to database types through SQLObject.
Line 133: Line 131:
 Python programs should never segfault, but it can happen if you trigger a bug in an extension module or the Python core itself. Since a segfault won't give you a Python traceback, it can be a bit daunting trying to debug these sort of problems. If you run into this sort of problem, tell the list. Python programs should never segfault, but it can happen if you trigger a bug in an extension module or the Python core itself. Since a segfault won't give you a Python traceback, it can be a bit daunting trying to debug these sort of problems. If you run into this sort of problem, tell the list.
Line 139: Line 137:
 Many Launchpad objects support `__getitem__`. For example, if you have a `Foo`, and want to support `Foo()['bar']`, you will implement `__getitem__` for class `Foo`. Often, this is used along with `GetitemNavigation` in your browser code to ensure smooth traversal.

 The `__getitem__` code itself should not, however, contain the magic that fetches whatever needs to be fetched. It should instead call another method that does so, explicitly saying what it is getting. So for example:
Many Launchpad objects support `__getitem__`. For example, if you have a `Foo`, and want to support `Foo()['bar']`, you will implement `__getitem__` for class `Foo`. Often, this is used along with `GetitemNavigation` in your browser code to ensure smooth traversal.

The `__getitem__` code itself should not, however, contain the magic that fetches whatever needs to be fetched. It should instead call another method that does so, explicitly saying what it is getting. So for example:
Line 145: Line 143:
@implementer(IFoo)
Line 146: Line 145:

    implements(IFoo)
Line 159: Line 156:
 Note that generally, a `__getitem__` method should give access to just one kind of thing. In the example above, it gives you access to versions with the given name. If your traversal needs to get two kinds of things, for example versions or changesets, then this is better put in the traversal code in the `FooNavigation` class than in the `__getitem__` code of the database class.

  ''I believe that the Launchpad coding style has evolved to frown upon unnecessary use of _``_magic_``_ methods. They make the code harder to grep and refactor. _``_getitem_``_ should only be used exceptionally when an object is meant to implement protocols that rely on it (list, dictionnary, string). Maybe this section should be updated to read something like "Don't".'' -- DavidAllouche <<DateTime(2007-11-14T16:43:09Z)>> 
Note that generally, a `__getitem__` method should give access to just one kind of thing. In the example above, it gives you access to versions with the given name. If your traversal needs to get two kinds of things, for example versions or changesets, then this is better put in the traversal code in the `FooNavigation` class than in the `__getitem__` code of the database class.

  ''I believe that the Launchpad coding style has evolved to frown upon unnecessary use of _``_magic_``_ methods. They make the code harder to grep and refactor. _``_getitem_``_ should only be used exceptionally when an object is meant to implement protocols that rely on it (list, dictionnary, string). Maybe this section should be updated to read something like "Don't".'' -- DavidAllouche <<DateTime(2007-11-14T16:43:09Z)>>
Line 165: Line 162:
 Properties should be cheap. Using a property can make accessing fields or calculated results easier, but programmers expect properties to be usable without consideration of the internal code in the property. As such, a property that calls expensive routines such as disk resources, examining database joins or the like will violate this expectation. This can lead to hard to analyse performance problems because its not clear what is going on unless you are very familiar with the code Properties should be cheap. Using a property can make accessing fields or calculated results easier, but programmers expect properties to be usable without consideration of the internal code in the property. As such, a property that calls expensive routines such as disk resources, examining database joins or the like will violate this expectation. This can lead to hard to analyse performance problems because its not clear what is going on unless you are very familiar with the code
Line 219: Line 216:
from canonical.launchpad.webapp.interfaces import ( from lp.services.webapp.interfaces import (
Line 241: Line 238:
from canonical.launchpad.webapp.interfaces import ( from lp.services.webapp.interfaces import (
Line 259: Line 256:
  the `ILaunchBag` in any database class.)    the `ILaunchBag` in any database class.)
Line 268: Line 265:
  You can't! Only attribute access can be protected and only the attribute name and the current user is available    You can't! Only attribute access can be protected and only the attribute name and the current user is available
Line 272: Line 269:
  That method can be appropriately protected using the current security infrastructure. Since this auxiallary method   That method can be appropriately protected using the current security infrastructure. Since this auxillary method
Line 276: Line 273:
  This will often happen with methods that needs to operate on bugs (since you usually don't want the operation to 
  be allowed if it's a private bug that the user doesn't have access to). 
  
  This will often happen with methods that needs to operate on bugs (since you usually don't want the operation to
  be allowed if it's a private bug that the user doesn't have access to).
Line 293: Line 290:
 As you know, persons and teams are meant to be interchangeable in Launchpad, but when it comes to mail notification the rules differ a bit (see TeamEmail for more information). In order to mask these rules, there's a helper function called contactEmailAddresses() in lib/canonical/launchpad/helpers.py that you should always use to get the contact address of a given person/team. (please note that this function will always return a Set of email addresses, which is perfectly suitable to be passed in to simple_sendmail())  As you know, persons and teams are meant to be interchangeable in Launchpad, but when it comes to mail notification the rules differ a bit (see TeamEmail for more information). In order to mask these rules, there's a helper function called get_contact_email_addresses() in lib/lp/services/mail/helpers.py that you should always use to get the contact address of a given person/team. (Please note that this function will always return a set of email addresses, which is perfectly suitable to be passed in to simple_sendmail().)
Line 297: Line 294:
=== When should I use SQLObjectEditView and SQLObjectAddView ===

 /!\ You should use LaunchpadFormView instead for add and general purpose views. There is a `LaunchpadEditFormView` specialed for edit form.
 Edit views (based on LaunchpadEditFormView) should be used only for edit forms.

 `SQLObjectEditView`, `SQLObjectAddView` and GeneralFormView are deprecated.

=== How do I use the General Form ===

 /!\ '''You don't!''' Use LaunchpadFormView instead.

 You need to inherit from the GeneralFormView in webapp/generalform.py. Your class should specify its schema, and the arguments / keyword arguments to its "process" method. In the process method you do whatever you want. You return a text string, which will be the message displayed to the user. If you want to redirect, then you either override the nextURL() method, or you set the value of the view's self._nextURL. Note that redirecting will lose any message you returned from self.process().

 Here's an example:

 {{{

  <browser:generalform
    name="+linkcve"
    for="canonical.launchpad.interfaces.IBugTask"
    schema="canonical.launchpad.interfaces.ICve"
    class="canonical.launchpad.browser.CveLinkView"
    fields="sequence"
    arguments="sequence"
    permission="launchpad.AnyPerson"
    template="../templates/bug-cve.pt">
  </browser:generalform>

from canonical.launchpad.webapp.generalform import GeneralFormView

class CveLinkView(GeneralFormView):
    """This view will be used for objects that can be linked to a CVE,
    currently that is only IBug.
    """

    def __init__(self, context, request):
        context = IBug(context)
        GeneralFormView.__init__(self, context, request)

    def process(self, sequence):
        cve = getUtility(ICveSet)[sequence]
        if cve is None:
            return '%s is not a known CVE sequence number.' % sequence
        user = getUtility(ILaunchBag).user
        self._nextURL = canonical_url(context)
        self.context.linkCVE(cve, user)
        return 'CVE-%s added to bug #%d' % (sequence, self.context.id)
}}}

 Notes:

  1. the ZCML is on IBugTask, because that is where this form will appear.
  1. the arguments and keyword arguments describe the interface to the self.process() method, which you must override. This method is where you do all your work.
  1. Note how the form just returns an error message in the case where the CVE cannot be located. This will be displayed for the user to see.
  1. If the CVE is located, it finds the user in the usual way, sets the next URL, executes the linkCVE method on the content object, and then returns a success message.
  1. Note that the success message will not be displayed for the user, because the browser will be redirecting. However, it is visible in the page tests. So you can test for both the redirect (303) and the success message. Very handy.
Line 419: Line 360:
=== How do I add items to launchpad.conf? ===

 This is done by changing the file lib/canonical/config/schema-lazr.conf. 
=== How do I add items to launchpad-lazr.conf? ===

 This is done by changing the file lib/lp/services/config/schema-lazr.conf.
Line 444: Line 385:
 * `development` — local development environment, used with `make run` and `make run_all`; launchpad.dev.  * `development` — local development environment, used with `make run` and `make run_all`; launchpad.test.
Line 449: Line 390:
=== How do I use the items listed in launchpad.conf? ===

 Once your items are added to the launchpad.conf file, you may use them as follows:

 {{{
>>> from canonical.config import config
=== How do I use the items listed in launchpad-lazr.conf? ===

 Once your items are added to the launchpad-lazr.conf file, you may use them as follows:

 {{{
>>> from lp.services.config import config
Line 456: Line 397:
>>> dbname = config.dbname       >>> dbname = config.dbname
Line 463: Line 404:
Since configuration is specified by a global object, `canonical.config.config`, tests for configuration-dependent will need to temporarily change configuration values.

Here's how to do it:
Use `lp.testing.TestCase.pushConfig`.

== Testing ==

=== What kind of tests should we use? ===

 See the TestsStyleGuide for the complete answer.

 Short answer is that we favor the use of doctest in `lib/lp/*/doc` for API documentation and PageTests for use-cases documentation. We use doctests and regular python unittest to complete the coverage.

=== How do I run just one doctest file, e.g. `lib/lp/*/doc/mytest.txt`? ===

 Use the `--test` argument to name it: {{{
bin/test --test=mytest.txt
}}}

=== What about running just one pagetest story, e.g. `lib/lp/*/stories/initial-bug-contacts`? ===

 {{{
bin/test --test=stories/initial-bug-contacts
}}}

=== What about running a standalone pagetest, e.g. `xx-bug-index.txt` ===

 Like this:

 {{{
bin/test --test=xx-bug-index
}}}

=== And if I want to execute all test except one? ===

 {{{
bin/test '!test_to_ignore'
}}}


=== How can I examine my test output with PDB? ===

  `bin/test`'s `-D` argument is ''everyone's'' friend.

  If your test raises any exceptions or failures, then the following will open a pdb shell right where the failure occurred:
Line 468: Line 448:
#!python
from canonical.config import config
from textwrap import dedent

name = 'test_some_feature'
config.push(name, dedent("""
    [some_section]
    some_key: some_value
    """)
# ...
config.pop(name)
}}}

Note that `config.push` will ''alter the global configuration of the current environment''. If you invoke it, you ''must call config.pop''. You can make sure this happens by:

 * Using try / finally blocks.
 * Placing the config.pop call in a TestCase.tearDown() method, if you are writing unit tests.
 * Using TestCase.addCleanup if your test inherits from `canonical.launchpad.tests.TestCase` (it probably should).

XXX: JonathanLange 2008-09-17: `pushConfig` in `codeimport.tests.test_dispatcher` should be moved into `TestCase` and this FAQ updated accordingly.

== Testing ==

=== What kind of tests should we use? ===

 See the TestsStyleGuide for the complete answer.

 Short answer is that we favor the use of doctest in `lib/canonical/launchpad/doc` for API documentation and PageTests for use-cases documentation. We use doctests and regular python unittest to complete the coverage.

=== How do I run just one doctest file, e.g. `lib/canonical/launchpad/doc/mytest.txt`? ===

 Use the `--test` argument to name it: {{{
test.py -f --test=mytest.txt
}}}

=== What about running just one pagetest story, e.g. `lib/canonical/launchpad/pagetests/initial-bug-contacts`? ===

 {{{
./test.py -f canonical pagetests.initial-bug-contacts
}}}

=== What about running a standalone pagetest, e.g. `xx-bug-index.txt` ===

 Like this:

 {{{
./test.py -f canonical xx-bug-index
}}}


 Note that you '''don't''' include the `.txt`.

=== And if I want to execute all test except one? ===

 {{{
./test.py '!test_to_ignore'
}}}


=== How can I examine my test output with PDB? ===

  `test.py`'s `-D` argument is ''everyone's'' friend.

  If your test raises any exceptions or failures, then the following will open a pdb shell right where the failure occurred:

  {{{
./test.py -D -vvt my.test.name
bin/test -D -vvt my.test.name
Line 542: Line 456:
./test.py --help bin/test --help
Line 547: Line 461:
 The ./test.py script has a --coverage option that will report on code coverage.   The bin/test script has a --coverage option that will report on code coverage.
Line 553: Line 467:
./test.py --layer=PageTestLayer bin/test --layer=PageTestLayer
Line 559: Line 473:
 
Line 562: Line 476:
 /!\ However! Resist the temptation to insert tests into the system doctests (lib/canonical/launchpad/doc/*.txt) that reduce their usefullness as documentation. Tests which confuse rather than clarify do not belong here. To a lesser extent, this also applies to other doctests too.  /!\ However! Resist the temptation to insert tests into the system doctests (lib/lp/*/doc/*.txt) that reduce their usefulness as documentation. Tests which confuse rather than clarify do not belong here. To a lesser extent, this also applies to other doctests too.
Line 566: Line 480:
 For DocFileSuite tests, such as the system documentation tests, you can pass in setUp and tearDown methods. You can stuff values into the namespace using the setUp method. See canonical/launchpad/ftests/test_system_documentation.py for examples, or the [[http://www.python.org/doc/current/lib/doctest-unittest-api.html|Python Reference guide]].  For DocFileSuite tests, such as the system documentation tests, you can pass in setUp and tearDown methods. You can stuff values into the namespace using the setUp method. See the [[http://www.python.org/doc/current/lib/doctest-unittest-api.html|Python Reference guide]].
Line 589: Line 503:
    >>> print line #doctest: -NORMALIZE_WHITESPACE<<BR>>     >>> print(line) #doctest: -NORMALIZE_WHITESPACE<<BR>>
Line 622: Line 536:
 Take a look at "lib/canonical/launchpad/pagetests/README.txt" for more information.
Line 626: Line 538:
 Build a fresh database, make the required changes, and then run 'make newsampledata' in the database/schema directory. This creates database/sampledata/newsampledata.sql. Rename this file to database/sampledata/current.sql, commit, push and merge.

 Note that as a side effect of the sampledata being automatically generated, you will often get difficult to resolve conflicts if you have modified the sample data and attempt to merge in another branch that has also modified it. To work around this, it is recommended that your sampledata changes are always maintained as list of statements in a .sql file so that you can easily reset your current.sql to the launchpad trunk and replay your changes against it (psql -d launchpad_dev -f mysampledatachanges.sql).
If you are making a schema change, [[PolicyAndProcess/DatabaseSchemaChangesProcess|please see this page instead]].

You shouldn't make changes to sample data unless you are submitting your changes to db-devel. You can save the state of the development site's database and replace the current-dev.sql so that everyone can see a working site.

 * {{{make schema}}} to create a clean db.
 * Do one of these next two.
   * If you have a patch (say, database/schema/patch-2208-99-0.sql), apply the patch to launchpad_ftest_playground and launchpad_dev.
      * {{{psql -f database/schema/patch-2208-99-0.sql launchpad_ftest_playground}}}
      * {{{psql -f database/schema/patch-2208-99-0.sql launchpad_dev}}}
   * Or, if you just want to change some sample data, using your browser or {{{make harness}}}
     * make the minimum changes needs to correct or add data to create the expected state.
     * If you you can undo you mistakes using the UI and harness for many actions, but sometimes you need to start over.
 * {{{make newsampledata}}}
   * This creates database/sampledata/newsampledata.sql and database/sampledata/newsampledata-dev.sql.
 * Review the changes and delete any additions to the karma table or backdate them to 2005 so that the site does not change as the data ages.
   * {{{diff database/sampledata/current.sql database/sampledata/newsampledata.sql}}}
   * {{{diff database/sampledata/current-dev.sql database/sampledata/newsampledata-dev.sql}}}
 * Move the new files into the right places.
   * {{{mv database/sampledata/newsampledata.sql database/sampledata/current.sql}}}
   * {{{mv database/sampledata/newsampledata-dev.sql database/sampledata/current-dev.sql}}}
 * {{{make schema}}} again.
 * Review the web site to verify your changes are correct.
 * {{{git commit}}}

Note that as a side effect of the sampledata being automatically generated, you will often get difficult to resolve conflicts if you have modified the sample data and attempt to merge in another branch that has also modified it. To work around this, it is recommended that your sampledata changes are always maintained as list of statements in a .sql file so that you can easily reset your current.sql to the launchpad trunk and replay your changes against it (psql -d launchpad_dev -f mysampledatachanges.sql).
Line 634: Line 568:
== Code Reviews ==

=== How do I use Meld with Bazaar to do code reviews? ===

 Simple way is to use the Bazaar difftools plugin, which will compare a
 Launchpad branch against the most recent version of launchpad the branch has
 been merged with. https://launchpad.net/bzr-difftools. Install that plugin, and
 install meld.

 The difftools plugin allows you to use many different external diff tools to
 compare branches. The basic form is `bzr diff --using=meld branch1 branch2`.
 The branches do not have to be on your local machine. If you try to compare remote
 branches difftools will take care of putting a working tree into a temp
 directory for meld to use, and then cleaning up the tree when you
 close the diff tool. Of course, if you compare local branches it is a bit
 faster and you can edit things in-place from inside of meld.

 You can define an alias in bazaar instead of typing `diff --using=meld` all the
 time. Just add something like this to your ~/.bazaar/bazaar.conf:
 {{{
[ALIASES]
mdiff = diff --using=meld
}}}

== Bazaar ==

=== How do I revert some revisions from my archive? ===

 If they are the most recent commits:
 `bzr uncommit`

 If they are not the most recent commits, then you need to do a reverse merge:
 `bzr merge -r newer..older`
 followed by a commit:
 `bzr commit`

=== I merged from another branch and did a lot of changes without first committing the merge. How can I undo only the merge? ===

==== If you are sure your changes don't touch the same code as the merge does, you can do this and it'll probably work: ====

 /!\ this should be updated - in bzr we can do a merge into a copy of the branch, then apply that in reverse mode.
 {{{
baz new-merges | tac | xargs -n1 baz replay --reverse
baz undo -o ,,my-changes
}}}

 Now, if you want to, you can do the merge again, commit and reaply your changes with:

 {{{
baz merge <whatever-branch-you-want>
baz commit -s "merge from <...>"
baz redo ,,my-changes
}}}

==== On the other hand, if your changes touch the same code as the merge, it's better to do the following ====

 /!\ still needs an update to bzr
 Assuming the worktree you have the merge and your changes is ~/wtree1, the tree-version of that worktree is ''me@canonical.com/cat--branch--ver'' and the branch you merged from is ''remote@host/cat--branch--ver'', you have to do this:


 {{{
cd ~/wtree1
FROM=$(baz new-merges | grep "remote@host/cat--branch--ver" | tail -n1)
cd ~/
baz get me@canonical.com/cat--branch--ver wtree2
cd wtree2
baz merge $FROM
baz commit -s "merge from <...>"
}}}

 Then you can easily get a changeset with only your changes by doing:

 {{{
baz delta ~/wtree1 ~/wtree2 ,,my-changes
}}}

 And commit only your changes:

 {{{
baz redo ,,my-changes
baz commit -s "..."
}}}

=== How can I get error failures from PQM by email? ===

 If you request a merge and it fails, PQM sends you back an email to the user address that requested the merge. If that machine is reachable from outside your network and is able to handle mail, you don't need to do anything. If you are behind a firewall or without an static IP address in your machine, you will need to masq those emails with a valid email address:

 Create or edit the file '''/etc/postfix/canonical''' and add a line like:

 {{{
#UNIX_USER EMAIL_ADDRESS
carlos carlos.perello@canonical.com
}}}

 Where UNIX_USER is the username you use in your system and EMAIL_ADDRESS is the email where you want to get all PQM mails.


 Now, update your system config to use that new file, edit '''/etc/postfix/main.cf''' and add a line like:

 {{{
canonical_maps = hash:/etc/postfix/canonical
}}}

 Finally, you will need to create a file that postfix is able to understand executing the following command, being inside the postfix directory:

 {{{

carlos@Gollum:/etc/postfix$postmap canonical

}}}


 Note, that this change will apply to all emails sent from that account not only to PQM ones, the From: header will point to the specified email address.
Line 765: Line 585:
 Get it reviewed and committed to rocketfuel as normal.   Get it reviewed and committed to rocketfuel as normal.
Line 767: Line 587:
 
Line 770: Line 590:
 The staging server is rebuilt daily using a copy of the production database and the HEAD of launchpad. It lets you see if your code changes actually work on the production system and performance is good. It lives at https://staging.launchpad.net/.
 The staging server is rebuilt daily using a copy of the production database and the HEAD of launchpad. It lets you see if your code changes actually work on the production system and performance is good. It lives at https://staging.launchpad.net/.

== Changing the launchpad dependency debs ==

The launchpad deb dependencies are in the ~launchpad PPA: https://launchpad.net/~launchpad/+archive/ppa. To change them, follow the instructions on the [[LaunchpadPpa#launchpad-dependencies|Launchpad PPA page]].

== How do I run scripts with the Python environment of Launchpad? ==

Use bin/py in your trunk, or whatever other branch you want to use. You might also want to use bin/ipy, if you like IPython.

== Pip ==

We use virtualenv and pip for our build system.

=== Where can I read about it? ===

You can read about how we set it up in [[https://git.launchpad.net/launchpad/tree/doc/pip.txt|doc/pip.txt]] in your trunk. This includes instructions for adding and upgrading distributions.

You can read more general information in [[https://pip.pypa.io/en/stable/|pip's documentation]].

=== How can I find out what we are using via pip? ===

For direct dependencies, look in setup.py (in the top directory of Launchpad). If it is in the "install_requires" argument, then we are getting it from pip.

For direct and indirect dependencies, look in constraints.txt (in the top directory of Launchpad). Note that this constrains more packages than we actually use.

=== How can I find out what we are using via sourcecode? ===

The canonical answer is to recursively find all symlinks in ''lib/'' and ''brzplugins/''. Here's a ''find'' incantation that will list the results: '''find lib brzplugins -lname '*/sourcecode/*' -ls''' .

You should also be able to look at utilities/sourcedeps.conf. This controls what is updated when sources are updated (for instance, via rocketfuel-get). However, it is maintained manually, so it could be behind the times if someone makes a mistake.

You do NOT answer this question by looking in the sourcecode directory. The sourcecode directory is typically a shared resource for all your branches, so it does not necessarily reflect what the current branch is using. It is also not cleaned out by any of our scripts.

=== Is the ultimate goal to completely get rid of sourcecode, so that all packages come from pip? ===

That was our original goal. We still want to shrink it to a very small size.

=== How do we make custom distributions? ===

See doc/pip.txt, mentioned above. Also see [[PolicyForDocumentingCustomDistributions|this proposed policy]].

== Working with our open-source components (lazr.*) ==

See HackingLazrLibraries.
Line 779: Line 642:
 * I have a view class that creates a Foo object, called FooAddView, and the view's context is a Bar object. Should I place the view together with other Foo-related views (in browser/foo.py) and the zcml declaration together with the views that share the same context (in zcml/bar.zcml)?   * I have a view class that creates a Foo object, called FooAddView, and the view's context is a Bar object. Should I place the view together with other Foo-related views (in browser/foo.py) and the zcml declaration together with the views that share the same context (in zcml/bar.zcml)?

See also: PythonStyleGuide and HackingLazrLibraries.

Want to navigate the source tree? Look at our API documentation.

Contents

  1. HTML, TAL, and CSS
  2. Python programming
    1. Which version of Python should I target?
    2. How should I format my docstrings?
    3. How should I use assertions in Launchpad code?
    4. What exceptions am I allowed to catch?
    5. What exception should I raise when something passed into an API isn't quite right?
    6. I have a self-posting form which doesn't display the updated values after it's submitted. What's wrong?
    7. I need to define a database class that includes a column of dbschema values. How do I do this?
    8. I have received a security proxied list from some API. I need to sort it, but the 'sort()' method is forbidden. What do I do?
    9. SQL Result Set Ordering
    10. How do I use a postgres stored-procedure/expression as the orderBy in SQLObject?
    11. How do I generate SQL commands safely?
    12. What date and time related types should I use?
    13. Python segfaulted. What should I do?
    14. I want an object to support __getitem__, what's the best style?
    15. Properties
  3. Storm
    1. How to retrieve a store ?
  4. Security, authentication
    1. How can I do get the current user in a database class?
    2. How can I protect a method based on one of its parameter?
  5. Email Notifications
    1. When I need to send a notification for a person/team, how do I know what email address(es) I have to send the notification to?
  6. Web UI
    1. How do I perform an action after an autogenerated edit form has been successfully submitted?
    2. How do I perform an action after an autogenerated add form has been successfully submitted?
    3. How can I redirect a user to a new object just created from an autogenerated add form?
    4. How do I format dates and times in page templates?
    5. How should I generate notices like "Added Bug #1234" to the top of the page?
  7. Launchpad API
    1. How do I add a new celebrity?
  8. Global Configuration
    1. How do I add items to launchpad-lazr.conf?
    2. How are these default values changed for specific environments?
    3. How do I use the items listed in launchpad-lazr.conf?
    4. How can I temporarily override configuration variables in tests?
  9. Testing
    1. What kind of tests should we use?
    2. How do I run just one doctest file, e.g. `lib/lp/*/doc/mytest.txt`?
    3. What about running just one pagetest story, e.g. `lib/lp/*/stories/initial-bug-contacts`?
    4. What about running a standalone pagetest, e.g. `xx-bug-index.txt`
    5. And if I want to execute all test except one?
    6. How can I examine my test output with PDB?
    7. Where can I get help on running tests?
    8. How can I check test coverage?
    9. How can I run only the tests for the page test layer?
    10. Where should I put my tests: in a `test_foo.py` module, or a `foo.txt` doctest file?
    11. How to I setup my tests namespace so I can remove unwanted import statements and other noise?
    12. Why is my page test failing mysteriously?
    13. I'm writing a pagetest in the standalone directory that changes some objects. Will my changes be visible to other pagetests in this directory?
    14. Why is not my page test failing when adding extra whitespace chars inside a textarea?
    15. What does the number in square brackets after a doctest name mean?
    16. How do I find a backtrace?
    17. Should I rely on the order of os.listdir?
    18. How do I find the tests that cover the code I've changed?
  10. Sample Data
    1. Where can I find a list of the sample users?
    2. How to I make changes to the sample Launchpad database?
    3. I've only made minor changes to the sample data, but the diff is huge!
    4. How do I set up connection multiplexing for my SSH session?
  11. Rollouts
    1. When will my code changes end up on production?
    2. I have an urgent bug fix
    3. What is the staging server?
  12. Changing the launchpad dependency debs
  13. How do I run scripts with the Python environment of Launchpad?
  14. Pip
    1. Where can I read about it?
    2. How can I find out what we are using via pip?
    3. How can I find out what we are using via sourcecode?
    4. Is the ultimate goal to completely get rid of sourcecode, so that all packages come from pip?
    5. How do we make custom distributions?
  15. Working with our open-source components (lazr.*)
  16. Where to go for other help
  17. Unanswered questions

HTML, TAL, and CSS

Python programming

Which version of Python should I target?

Currently, Launchpad requires Python 3.5. You may use features that require Python 3.5, as long as you abide by the PythonStyleGuide.

How should I format my docstrings?

First of all, thank you for writing docstrings. They make the code much easier to follow for the people that didn't originally write it. To answer the question, you have the following options.

  • A single short sentence.
  • A single short sentence, blank line, further explanation.
  • A single short sentence, blank line, rst-style explanation of arguments and return value.
  • A single short sentence, blank line, further explanation with rst-style explanation of arguments and return value.

    You may include a short doctest in the further explanation. See the examples in the Epydoc documentation. We're using the rst or ReStructuredText format, and not using the Javadoc or Epytext formats.

    See also: PEP-8, Style Guide for Python Code, Docstring Conventions. Note that we're not using the full expressiveness of ReStructuredText, so PEP-287 doesn't apply.

How should I use assertions in Launchpad code?

What exceptions am I allowed to catch?

See ExceptionGuidelines.

What exception should I raise when something passed into an API isn't quite right?

In short, never raise ValueError, NameError or TypeError, and avoid subclassing these exceptions as well. The full instructions are at ExceptionGuidelines.

In the case of NotFoundError, if you are going to catch this specific error in some other code, and then take some corrective action or some logging action, then seriously consider creating your own subclass. This allows your code to handle exactly the situation that you expect, and not be tricked into handling NotFoundErrors raised by code several levels in.

When writing docstrings, always think whether it makes things clearer to say which exceptions will be raised due to inappropriate values being passed in.

I have a self-posting form which doesn't display the updated values after it's submitted. What's wrong?

For now, all self-posting forms have to call lp.services.database.sqlbase.flush_database_updates() after processing the form.

I need to define a database class that includes a column of dbschema values. How do I do this?

Use an EnumCol. See the EnumCol wiki page for how it works.

I have received a security proxied list from some API. I need to sort it, but the 'sort()' method is forbidden. What do I do?

When you get a list from a security proxied object, that list is protected against being altered. This is important, because you don't know who else might have a reference to that list.

When programming in Python generally, it is a good idea to make a copy of a list you get from somewhere before altering it. The security proxies just enforce this good practice.

You can make a copy of the list by using the 'list' constructor. Here is an example, using the launchpad API.

   1   members = getUtility(ITeamParticipationSet).getAllMembers()
   2   members = list(members)  # Get a mutable list.
   3   members.sort()

You can also use the sorted builtin to do this.

   1   members = sorted(getUtility(ITeamParticipationSet).getAllMembers())

SQL Result Set Ordering

If the ordering of an SQL result set is not fully constrained, then your tests should not be dependent on the natural ordering of results in the sample data.

If Launchpad does not depend on the ordering of a particular result set, then that result set should be sorted within the test so that it will pass for any ordering.

As a general rule, the result sets whose order we want to test are the ones that are displayed to the user. These should use an ordering that makes sense to the user, and the tests should ensure that happens.

How do I use a postgres stored-procedure/expression as the orderBy in SQLObject?

You have to wrap it in an SQLConstant:

from sqlobject.sqlbuilder import SQLConstant, DESC

Person.select(orderBy=SQLConstant("person_sort_key(displayname, name)"))

Ticket.select("fti @@ ftq('firefox')", orderBy=DESC(SQLConstant("rank(fti, ftq('firefox'))")))

How do I generate SQL commands safely?

The safest way is to ensure that all data is passed in as parameters, the way the DB-API intended it:

   1  cur = con.cursor()
   2  cur.execute("select id,name from Person where displayname=%(name)s", {'name': 'Stuart Bishop'})
   3  results = cur.fetchall()

If you need to embed your data in the SQL query itself, there is only one rule you need to remember - quote your data. Failing to do this opens up the system to an SQL Injection attack, one of the more common and widely known security holes and also one of the more destructive. Don't attempt to write your own quote method - you will probably get it wrong and end up with the DBA removing precious parts of your anatomy. The only two formats you can use are %d and %s, and %s should always be escaped, *no exceptions!*

   1  from lp.services.database.sqlbase import quote
   2  cur = con.cursor()
   3  cur.execute("select id,name from Person where displayname=%s" % quote("Stuart Bishop"))
   4  results = cur.fetchall()
   5 
   6  cur.execute("select * from Person where name=%s" % quote(
   7    "'; drop table person cascade; insert into person(name) values ('hahaloser')"))

(the second command in the previous example demonstrates a simple argument that might be passed in by an attacker).

See DatetimeUsageGuide for information on what types to use, and how the Python datetime types relate to database types through SQLObject.

Python segfaulted. What should I do?

Python programs should never segfault, but it can happen if you trigger a bug in an extension module or the Python core itself. Since a segfault won't give you a Python traceback, it can be a bit daunting trying to debug these sort of problems. If you run into this sort of problem, tell the list.

  • See DebuggingWithGdb for some tips on how to narrow down where a segfault bug like this is occurring -- in some cases you can even get a Python stack trace for this sort of problem.

I want an object to support __getitem__, what's the best style?

Many Launchpad objects support __getitem__. For example, if you have a Foo, and want to support Foo()['bar'], you will implement __getitem__ for class Foo. Often, this is used along with GetitemNavigation in your browser code to ensure smooth traversal.

The __getitem__ code itself should not, however, contain the magic that fetches whatever needs to be fetched. It should instead call another method that does so, explicitly saying what it is getting. So for example:

  •    1 @implementer(IFoo)
       2 class FooContentClass:
       3 
       4     def __getitem__(self, name):
       5         """See IFoo."""
       6         return self.getVersion(name)
       7 
       8     def getVersion(self, name):
       9         """See IFoo."""
      10         # blah blah blah
      11         return version
    

Note that generally, a __getitem__ method should give access to just one kind of thing. In the example above, it gives you access to versions with the given name. If your traversal needs to get two kinds of things, for example versions or changesets, then this is better put in the traversal code in the FooNavigation class than in the __getitem__ code of the database class.

  • I believe that the Launchpad coding style has evolved to frown upon unnecessary use of __magic__ methods. They make the code harder to grep and refactor. __getitem__ should only be used exceptionally when an object is meant to implement protocols that rely on it (list, dictionnary, string). Maybe this section should be updated to read something like "Don't". -- DavidAllouche 2007-11-14 16:43:09

Properties

Properties should be cheap. Using a property can make accessing fields or calculated results easier, but programmers expect properties to be usable without consideration of the internal code in the property. As such, a property that calls expensive routines such as disk resources, examining database joins or the like will violate this expectation. This can lead to hard to analyse performance problems because its not clear what is going on unless you are very familiar with the code

  • Our code routinely contradicts this guideline. I remember I had issues in the past with TALES traversal when trying to use methods, and had to use properties instead. We have decorators such as @cachedproperty to help with the performance issues. Someone who knows what he talks about should update this FAQ to match reality. -- DavidAllouche 2007-11-14 16:50:06

Properties should always be used instead of __call__() semantics in TALES expressions. The rule is that in view classes, we don't do this:

    def foo(self):
        ...

We always do this:

    @property
    def foo(self):
        ...

Storm

Questions about Storm usage.

StormMigrationGuide document is highly recommended.

How to retrieve a store ?

There are two ways of retrieving a storm 'store', before issuing a query using native syntax.

The first format retrieves the Store being used by another object. Use this method when you don't need to make changes, but want your objects to interact nicely with objects from an unknown Store (such as a methods parameters):

   1 from storm.store import Store
   2 
   3 store = Store.of(object)
   4 
   5 result = store.find(Person, Person.name == 'zeca')

You can also explicitly specify what Store you want to use. You get to choose the realm (Launchpad main, auth database) and the flavor (master or slave). If you are retrieving objects that will need to be updated, you need to use the master. If you are doing a search and we don't mind querying data a few seconds out of date, you should use the slave.

   1 from lp.services.webapp.interfaces import (
   2     IStoreSelector, MAIN_STORE, AUTH_STORE,
   3     MASTER_FLAVOR, SLAVE_FLAVOR)
   4 
   5 master_store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
   6 
   7 master_obj = store.find(Person, Person.name == 'zeca')
   8 
   9 slave_store = getUtility(IStoreSelector).get(MAIN_STORE, SLAVE_FLAVOR)
  10 
  11 slave_obj = store.find(Person, Person.name == 'zeca')

If you don't need to update, but require up-to-date data, you should use the default flavor. (eg. most views - the object you are viewing might just have been created). This will retrieve the master unless the load balancer is sure all changes made by the current client have propagated to the replica databases.

   1 from lp.services.webapp.interfaces import (
   2     IStoreSelector, MAIN_STORE, AUTH_STORE, DEFAULT_FLAVOR)
   3 
   4 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
   5 
   6 result = store.find(Person, Person.name == 'zeca')

See more details about this in https://lists.ubuntu.com/mailman/private/launchpad/2008-August/031418.html

Security, authentication

How can I do get the current user in a database class?

  • You need to pass it in one of the parameter's method. (You shouldn't use the ILaunchBag for this. In fact, you shouldn't use the ILaunchBag in any database class.) The principle is that the database code must not rely on implicit state, and by that is meant state not present in the database object's data nor

    in the arguments passed to the method call. Using ILaunchBag or check_permission would use this kind of implicit state.

How can I protect a method based on one of its parameter?

  • You can't! Only attribute access can be protected and only the attribute name and the current user is available when that check is made. But there is a common pattern you can use: call in that method another method on the object passed as parameter. That method can be appropriately protected using the current security infrastructure. Since this auxillary method

    is part of an object-collaboration scenario, it's usually a good idea to start these methods with the notify verb. The method is notifying the other object that a collaboration is taking place. This will often happen with methods that needs to operate on bugs (since you usually don't want the operation to be allowed if it's a private bug that the user doesn't have access to). Example:

    • def linkBug(self, bug):
         # If this is a private bug that the user doesn't have access, it
         # will raise an Unauthorized error.
         bug.notifyLinkBug(self)

Email Notifications

When I need to send a notification for a person/team, how do I know what email address(es) I have to send the notification to?

  • As you know, persons and teams are meant to be interchangeable in Launchpad, but when it comes to mail notification the rules differ a bit (see TeamEmail for more information). In order to mask these rules, there's a helper function called get_contact_email_addresses() in lib/lp/services/mail/helpers.py that you should always use to get the contact address of a given person/team. (Please note that this function will always return a set of email addresses, which is perfectly suitable to be passed in to simple_sendmail().)

Web UI

How do I perform an action after an autogenerated edit form has been successfully submitted?

  • You need to write a view's class for this form, if you don't have one already. In your view's class, add

    a method changed().

       1     def changed(self):
       2         # This method is called after changes have been made.
    
    You can use this hook to add a redirect, or to execute some logging, for example.

How do I perform an action after an autogenerated add form has been successfully submitted?

  • You need to write a view's class for this form, if you don't have one already. In your view's class, add

    a method createAndAdd().

       1     def createAndAdd(self, data):
       2         # This method is called with the data from the form.
    
    You can use this hook to create new objects based on the input from the user.

How can I redirect a user to a new object just created from an autogenerated add form?

  • You need to write a view's class for this form, if you don't have one already. In your view's class, add

    a method nextURL().

       1     def nextURL(self):
       2         # This method returns the URL where the user should be redirected.
    

How do I format dates and times in page templates?

  • Let's use some object's datecreated attribute as an example.

    To format a date, use tal:content="context/datecreated/fmt:date

    To format a time, use tal:content="context/datecreated/fmt:time

    To format a date and time, use tal:content="context/datecreated/fmt:datetime

How should I generate notices like "Added Bug #1234" to the top of the page?

  •    1 response.addInfoNotification('Added <b>Bug #%(bug_id)d</d>', bug_id=bug.id)
    

    There are other notification levels (Debug, Info, Notice, Warning, Error), as per BrowserNotificationMessages

Launchpad API

How do I add a new celebrity?

Global Configuration

How do I add items to launchpad-lazr.conf?

  • This is done by changing the file lib/lp/services/config/schema-lazr.conf.
  • Items should be created with the following syntax:
    • # Comment describing the item.
      key_name: default_value

key_name must be a valid Python identifier.

  • Subsections should be created with the following syntax:
    • [section_name]
      key_name: ...

section_name must be a valid Python identifier.

How are these default values changed for specific environments?

The default configuration values are overridden in /configs/<environment>/launchpad-lazr.conf. Notable environments include:

  • production — the production environment; launchpad.net.

  • staging — the staging environment; staging.launchpad.net.

  • development — local development environment, used with make run and make run_all; launchpad.test.

  • testrunner — the environment used when running tests.

The syntax for overriding configuration values is the same as the syntax for defining them.

How do I use the items listed in launchpad-lazr.conf?

  • Once your items are added to the launchpad-lazr.conf file, you may use them as follows:
    >>> from lp.services.config import config
    >>> # We grab dbname from the default section
    >>> dbname = config.dbname
    >>> # We grab the dbuser from the gina subsection
    >>> dbuser = config.gina.dbuser

How can I temporarily override configuration variables in tests?

Use lp.testing.TestCase.pushConfig.

Testing

What kind of tests should we use?

  • See the TestsStyleGuide for the complete answer.

    Short answer is that we favor the use of doctest in lib/lp/*/doc for API documentation and PageTests for use-cases documentation. We use doctests and regular python unittest to complete the coverage.

How do I run just one doctest file, e.g. `lib/lp/*/doc/mytest.txt`?

  • Use the --test argument to name it:

    bin/test --test=mytest.txt

What about running just one pagetest story, e.g. `lib/lp/*/stories/initial-bug-contacts`?

  • bin/test --test=stories/initial-bug-contacts

What about running a standalone pagetest, e.g. `xx-bug-index.txt`

  • Like this:
    bin/test --test=xx-bug-index

And if I want to execute all test except one?

  • bin/test '!test_to_ignore'

How can I examine my test output with PDB?

  • bin/test's -D argument is everyone's friend. If your test raises any exceptions or failures, then the following will open a pdb shell right where the failure occurred:

    bin/test -D -vvt my.test.name

Where can I get help on running tests?

  • Try this:
    bin/test --help

How can I check test coverage?

  • The bin/test script has a --coverage option that will report on code coverage.

How can I run only the tests for the page test layer?

  • bin/test --layer=PageTestLayer

Where should I put my tests: in a `test_foo.py` module, or a `foo.txt` doctest file?

  • You should prefer doctests. A good rule of thumb is that test_*.py modules are best for tests that aren't useful for documentation, but instead for increasing test coverage to obscure or hard-to-reach code paths. It is very easy to write test code that says "check foo does bar", without explaining why. Doctests tend to trick the author into explaining why.

    /!\ However! Resist the temptation to insert tests into the system doctests (lib/lp/*/doc/*.txt) that reduce their usefulness as documentation. Tests which confuse rather than clarify do not belong here. To a lesser extent, this also applies to other doctests too.

How to I setup my tests namespace so I can remove unwanted import statements and other noise?

  • For DocFileSuite tests, such as the system documentation tests, you can pass in setUp and tearDown methods. You can stuff values into the namespace using the setUp method. See the Python Reference guide.

       1 from zope.component import getUtility
       2 def setUp(test):
       3     test.globs['getUtility'] = getUtility
    

Why is my page test failing mysteriously?

  • This is often due to a bug in the doctest code that means that ellipses ("...") don't match blank lines ("<BLANKLINE>"). Inserting blank lines in the right parts of the page test should fix it. If you are running a single test and getting odd database failures, changes are you haven't run make schema. When running a single test the database setup step is skipped, and you need to make sure you've done it before.

I'm writing a pagetest in the standalone directory that changes some objects. Will my changes be visible to other pagetests in this directory?

  • No. The database is reset between each standalone pagetest.

Why is not my page test failing when adding extra whitespace chars inside a textarea?

  • Because by default, the page test ignore them so you don't need to take care of the indentation. Sometimes, the indentation matters (inside <pre> and <textarea> tags) and if you want to test those, you will need to append to the test '#doctest: -NORMALIZE_WHITESPACE', for instance:

    • >>> print(line) #doctest: -NORMALIZE_WHITESPACE
      foo

What does the number in square brackets after a doctest name mean?

  • When you get a warning or an error from a doctest, or you see it in a traceback, you'll see the name written with a number in square brackets, like this:
    <doctest gpg-encryption.txt[16]>

    The number, in the text above it is "16", is the index of the "doctest example" in that file. What this means is that the first >>> in the file is example zero. The next >>> is example one, and so on. So [16] refers to the seventeenth line starting with >>> in the doctest file gpg-encrpytion.txt.

How do I find a backtrace?

  • Backtraces are kept on chinstrap in the /srv/gangotri-logs directory.

Should I rely on the order of os.listdir?

  • No. The order in which results are returned by os.listdir is not defined. Wrap your calls with sorted() if the order is important to passing the test.

How do I find the tests that cover the code I've changed?

Although you've written the tests first before changing code, a lot of code is also exercised by many other tests and it's not immediately obvious which those might be without running the whole test suite. There is an experimental tool to help with this at TestsFromChanges.

Sample Data

Sampledata should be innocuous, such that if it is viewed by people outside Canonical, it will not be embarrassing or reveal company-confidential information.

Where can I find a list of the sample users?

  • As an aid to testing, the sample database provides a set of sample users with differing memberships and authorization levels.

How to I make changes to the sample Launchpad database?

If you are making a schema change, please see this page instead.

You shouldn't make changes to sample data unless you are submitting your changes to db-devel. You can save the state of the development site's database and replace the current-dev.sql so that everyone can see a working site.

  • make schema to create a clean db.

  • Do one of these next two.
    • If you have a patch (say, database/schema/patch-2208-99-0.sql), apply the patch to launchpad_ftest_playground and launchpad_dev.
      • psql -f database/schema/patch-2208-99-0.sql launchpad_ftest_playground

      • psql -f database/schema/patch-2208-99-0.sql launchpad_dev

    • Or, if you just want to change some sample data, using your browser or make harness

      • make the minimum changes needs to correct or add data to create the expected state.
      • If you you can undo you mistakes using the UI and harness for many actions, but sometimes you need to start over.
  • make newsampledata

    • This creates database/sampledata/newsampledata.sql and database/sampledata/newsampledata-dev.sql.
  • Review the changes and delete any additions to the karma table or backdate them to 2005 so that the site does not change as the data ages.
    • diff database/sampledata/current.sql database/sampledata/newsampledata.sql

    • diff database/sampledata/current-dev.sql database/sampledata/newsampledata-dev.sql

  • Move the new files into the right places.
    • mv database/sampledata/newsampledata.sql database/sampledata/current.sql

    • mv database/sampledata/newsampledata-dev.sql database/sampledata/current-dev.sql

  • make schema again.

  • Review the web site to verify your changes are correct.
  • git commit

Note that as a side effect of the sampledata being automatically generated, you will often get difficult to resolve conflicts if you have modified the sample data and attempt to merge in another branch that has also modified it. To work around this, it is recommended that your sampledata changes are always maintained as list of statements in a .sql file so that you can easily reset your current.sql to the launchpad trunk and replay your changes against it (psql -d launchpad_dev -f mysampledatachanges.sql).

I've only made minor changes to the sample data, but the diff is huge!

  • This often happens when you have modified the schema. Don't worry - if the tests pass, then the modified sample data is good.

How do I set up connection multiplexing for my SSH session?

Rollouts

See ReleaseCycles for more information.

When will my code changes end up on production?

  • PQM is frozen sometime on, or shortly after, Friday of week 3. Authorized Release Critical changes are then processed until Wednesday afternoon when the roll-out usually happens.

    Bug fixes discovered in this release on the staging server may be cherry picked into the release. Further discussion can be found on the StagingServer page.

I have an urgent bug fix

  • Get it reviewed and committed to rocketfuel as normal. If it's during Week 4, contact the release manager (usually kiko). Otherwise, follow the PolicyandProcess/EChangePolicy

What is the staging server?

  • The staging server is rebuilt daily using a copy of the production database and the HEAD of launchpad. It lets you see if your code changes actually work on the production system and performance is good. It lives at https://staging.launchpad.net/.

Changing the launchpad dependency debs

The launchpad deb dependencies are in the ~launchpad PPA: https://launchpad.net/~launchpad/+archive/ppa. To change them, follow the instructions on the Launchpad PPA page.

How do I run scripts with the Python environment of Launchpad?

Use bin/py in your trunk, or whatever other branch you want to use. You might also want to use bin/ipy, if you like IPython.

Pip

We use virtualenv and pip for our build system.

Where can I read about it?

You can read about how we set it up in doc/pip.txt in your trunk. This includes instructions for adding and upgrading distributions.

You can read more general information in pip's documentation.

How can I find out what we are using via pip?

For direct dependencies, look in setup.py (in the top directory of Launchpad). If it is in the "install_requires" argument, then we are getting it from pip.

For direct and indirect dependencies, look in constraints.txt (in the top directory of Launchpad). Note that this constrains more packages than we actually use.

How can I find out what we are using via sourcecode?

The canonical answer is to recursively find all symlinks in lib/ and brzplugins/. Here's a find incantation that will list the results: find lib brzplugins -lname '*/sourcecode/*' -ls .

You should also be able to look at utilities/sourcedeps.conf. This controls what is updated when sources are updated (for instance, via rocketfuel-get). However, it is maintained manually, so it could be behind the times if someone makes a mistake.

You do NOT answer this question by looking in the sourcecode directory. The sourcecode directory is typically a shared resource for all your branches, so it does not necessarily reflect what the current branch is using. It is also not cleaned out by any of our scripts.

Is the ultimate goal to completely get rid of sourcecode, so that all packages come from pip?

That was our original goal. We still want to shrink it to a very small size.

How do we make custom distributions?

See doc/pip.txt, mentioned above. Also see this proposed policy.

Working with our open-source components (lazr.*)

See HackingLazrLibraries.

Where to go for other help

If you have encountered a problem, maybe someone else has and has already documented it on the SolutionsLog. Go have a look! If have a problem and you find a solution, document it there!

Unanswered questions

  • I have a view class that creates a Foo object, called FooAddView, and the view's context is a Bar object. Should I place the view together with other Foo-related views (in browser/foo.py) and the zcml declaration together with the views that share the same context (in zcml/bar.zcml)?

Hacking (last edited 2021-11-11 09:54:30 by cjwatson)