Hacking

Not logged in - Log In / Register

Revision 21 as of 2011-06-21 10:40:26

Clear message

See also: PythonStyleGuide and HackingLazrLibraries.

Want to navigate the source tree? Look at our API documentation.

Contents

  1. HTML, TAL, and CSS
  2. Python programming
    1. Which version of Python should I target?
    2. How should I format my docstrings?
    3. How should I use assertions in Launchpad code?
    4. What exceptions am I allowed to catch?
    5. What exception should I raise when something passed into an API isn't quite right?
    6. I have a self-posting form which doesn't display the updated values after it's submitted. What's wrong?
    7. I need to define a database class that includes a column of dbschema values. How do I do this?
    8. I have received a security proxied list from some API. I need to sort it, but the 'sort()' method is forbidden. What do I do?
    9. SQL Result Set Ordering
    10. How do I use a postgres stored-procedure/expression as the orderBy in SQLObject?
    11. How do I generate SQL commands safely?
    12. What date and time related types should I use?
    13. Python segfaulted. What should I do?
    14. I want an object to support __getitem__, what's the best style?
    15. Properties
  3. Storm
    1. How to retrieve a store ?
  4. Security, authentication
    1. How can I do get the current user in a database class?
    2. How can I protect a method based on one of its parameter?
  5. Email Notifications
    1. When I need to send a notification for a person/team, how do I know what email address(es) I have to send the notification to?
  6. Web UI
    1. When should I use SQLObjectEditView and SQLObjectAddView
    2. How do I use the General Form
    3. How do I perform an action after an autogenerated edit form has been successfully submitted?
    4. How do I perform an action after an autogenerated add form has been successfully submitted?
    5. How can I redirect a user to a new object just created from an autogenerated add form?
    6. How do I format dates and times in page templates?
    7. How should I generate notices like "Added Bug #1234" to the top of the page?
  7. Launchpad API
    1. How do I add a new celebrity?
  8. Global Configuration
    1. How do I add items to launchpad.conf?
    2. How are these default values changed for specific environments?
    3. How do I use the items listed in launchpad.conf?
    4. How can I temporarily override configuration variables in tests?
  9. Testing
    1. What kind of tests should we use?
    2. How do I run just one doctest file, e.g. `lib/canonical/launchpad/doc/mytest.txt`?
    3. What about running just one pagetest story, e.g. `lib/canonical/launchpad/pagetests/initial-bug-contacts`?
    4. What about running a standalone pagetest, e.g. `xx-bug-index.txt`
    5. And if I want to execute all test except one?
    6. How can I examine my test output with PDB?
    7. Where can I get help on running tests?
    8. How can I check test coverage?
    9. How can I run only the tests for the page test layer?
    10. Where should I put my tests: in a `test_foo.py` module, or a `foo.txt` doctest file?
    11. How to I setup my tests namespace so I can remove unwanted import statements and other noise?
    12. Why is my page test failing mysteriously?
    13. I'm writing a pagetest in the standalone directory that changes some objects. Will my changes be visible to other pagetests in this directory?
    14. Why is not my page test failing when adding extra whitespace chars inside a textarea?
    15. What does the number in square brackets after a doctest name mean?
    16. How do I find a backtrace?
    17. Should I rely on the order of os.listdir?
    18. How do I find the tests that cover the code I've changed?
  10. Sample Data
    1. Where can I find a list of the sample users?
    2. How to I make changes to the sample Launchpad database?
    3. I've only made minor changes to the sample data, but the diff is huge!
  11. Code Reviews
    1. How do I use Meld with Bazaar to do code reviews?
  12. Bazaar
    1. How do I revert some revisions from my archive?
    2. I merged from another branch and did a lot of changes without first committing the merge. How can I undo only the merge?
      1. If you are sure your changes don't touch the same code as the merge does, you can do this and it'll probably work:
      2. On the other hand, if your changes touch the same code as the merge, it's better to do the following
    3. How can I get error failures from PQM by email?
    4. How do I set up connection multiplexing for my SSH session?
  13. Rollouts
    1. When will my code changes end up on production?
    2. I have an urgent bug fix
    3. What is the staging server?
  14. Changing the launchpad dependency debs
  15. How do I run scripts with the Python environment of Launchpad?
  16. Buildout
    1. Where can I read about it?
    2. How can I find out what we are using via buildout?
    3. How can I find out what we are using via sourcecode?
    4. Is the ultimate goal to completely get rid of sourcecode, so that all packages come from buildout?
    5. How do we make custom distributions?
  17. Working with our open-source components (lazr.*)
  18. Where to go for other help
  19. Unanswered questions

HTML, TAL, and CSS

Python programming

Which version of Python should I target?

Currently, Launchpad and Bazaar require Python 2.5. You may use features that require Python 2.5, as long as you abide by the PythonStyleGuide.

How should I format my docstrings?

First of all, thank you for writing docstrings. They make the code much easier to follow for the people that didn't originally write it. To answer the question, you have the following options.

How should I use assertions in Launchpad code?

What exceptions am I allowed to catch?

See ExceptionGuidelines.

What exception should I raise when something passed into an API isn't quite right?

In short, never raise ValueError, NameError or TypeError, and avoid subclassing these exceptions as well. The full instructions are at ExceptionGuidelines.

In the case of NotFoundError, if you are going to catch this specific error in some other code, and then take some corrective action or some logging action, then seriously consider creating your own subclass. This allows your code to handle exactly the situation that you expect, and not be tricked into handling NotFoundErrors raised by code several levels in.

When writing docstrings, always think whether it makes things clearer to say which exceptions will be raised due to inappropriate values being passed in.

I have a self-posting form which doesn't display the updated values after it's submitted. What's wrong?

For now, all self-posting forms have to call canonical.database.sqlbase.flush_database_updates() after processing the form.

I need to define a database class that includes a column of dbschema values. How do I do this?

Use an EnumCol. See the EnumCol wiki page for how it works.

I have received a security proxied list from some API. I need to sort it, but the 'sort()' method is forbidden. What do I do?

When you get a list from a security proxied object, that list is protected against being altered. This is important, because you don't know who else might have a reference to that list.

When programming in Python generally, it is a good idea to make a copy of a list you get from somewhere before altering it. The security proxies just enforce this good practice.

You can make a copy of the list by using the 'list' constructor. Here is an example, using the launchpad API.

   1   members = getUtility(ITeamParticipationSet).getAllMembers()
   2   members = list(members)  # Get a mutable list.
   3   members.sort()

You can also use the sorted builtin to do this.

   1   members = sorted(getUtility(ITeamParticipationSet).getAllMembers())

SQL Result Set Ordering

If the ordering of an SQL result set is not fully constrained, then your tests should not be dependent on the natural ordering of results in the sample data.

If Launchpad does not depend on the ordering of a particular result set, then that result set should be sorted within the test so that it will pass for any ordering.

As a general rule, the result sets whose order we want to test are the ones that are displayed to the user. These should use an ordering that makes sense to the user, and the tests should ensure that happens.

For code that uses SQLObject, result sets will be randomised while conforming to the specified ORDER BY clause. This makes it difficult to write tests that depend on the natural ordering.

In contrast, code that does raw SQL won't have a randomised result set, so the natural order gets exposed to the user. The same rules of thumb apply when testing such code, but extra care should be taken not to rely on natural ordering.

How do I use a postgres stored-procedure/expression as the orderBy in SQLObject?

You have to wrap it in an SQLConstant:

from sqlobject.sqlbuilder import SQLConstant, DESC

Person.select(orderBy=SQLConstant("person_sort_key(displayname, name)"))

Ticket.select("fti @@ ftq('firefox')", orderBy=DESC(SQLConstant("rank(fti, ftq('firefox'))")))

How do I generate SQL commands safely?

The safest way is to ensure that all data is passed in as parameters, the way the DB-API intended it:

   1  cur = con.cursor()
   2  cur.execute("select id,name from Person where displayname=%(name)s", {'name': 'Stuart Bishop'})
   3  results = cur.fetchall()

If you need to embed your data in the SQL query itself, there is only one rule you need to remember - quote your data. Failing to do this opens up the system to an SQL Injection attack, one of the more common and widely known security holes and also one of the more destructive. Don't attempt to write your own quote method - you will probably get it wrong and end up with the DBA removing precious parts of your anatomy. The only two formats you can use are %d and %s, and %s should always be escaped, *no exceptions!*

   1  from canonical.database.sqlbase import quote
   2  cur = con.cursor()
   3  cur.execute("select id,name from Person where displayname=%s" % quote("Stuart Bishop"))
   4  results = cur.fetchall()
   5 
   6  cur.execute("select * from Person where name=%s" % quote(
   7    "'; drop table person cascade; insert into person(name) values ('hahaloser')"))

(the second command in the previous example demonstrates a simple argument that might be passed in by an attacker).

See DatetimeUsageGuide for information on what types to use, and how the Python datetime types relate to database types through SQLObject.

Python segfaulted. What should I do?

Python programs should never segfault, but it can happen if you trigger a bug in an extension module or the Python core itself. Since a segfault won't give you a Python traceback, it can be a bit daunting trying to debug these sort of problems. If you run into this sort of problem, tell the list.

I want an object to support __getitem__, what's the best style?

Many Launchpad objects support __getitem__. For example, if you have a Foo, and want to support Foo()['bar'], you will implement __getitem__ for class Foo. Often, this is used along with GetitemNavigation in your browser code to ensure smooth traversal.

The __getitem__ code itself should not, however, contain the magic that fetches whatever needs to be fetched. It should instead call another method that does so, explicitly saying what it is getting. So for example:

Note that generally, a __getitem__ method should give access to just one kind of thing. In the example above, it gives you access to versions with the given name. If your traversal needs to get two kinds of things, for example versions or changesets, then this is better put in the traversal code in the FooNavigation class than in the __getitem__ code of the database class.

Properties

Properties should be cheap. Using a property can make accessing fields or calculated results easier, but programmers expect properties to be usable without consideration of the internal code in the property. As such, a property that calls expensive routines such as disk resources, examining database joins or the like will violate this expectation. This can lead to hard to analyse performance problems because its not clear what is going on unless you are very familiar with the code

Properties should always be used instead of __call__() semantics in TALES expressions. The rule is that in view classes, we don't do this:

    def foo(self):
        ...

We always do this:

    @property
    def foo(self):
        ...

Storm

Questions about Storm usage.

StormMigrationGuide document is highly recommended.

How to retrieve a store ?

There are two ways of retrieving a storm 'store', before issuing a query using native syntax.

The first format retrieves the Store being used by another object. Use this method when you don't need to make changes, but want your objects to interact nicely with objects from an unknown Store (such as a methods parameters):

   1 from storm.store import Store
   2 
   3 store = Store.of(object)
   4 
   5 result = store.find(Person, Person.name == 'zeca')

You can also explicitly specify what Store you want to use. You get to choose the realm (Launchpad main, auth database) and the flavor (master or slave). If you are retrieving objects that will need to be updated, you need to use the master. If you are doing a search and we don't mind querying data a few seconds out of date, you should use the slave.

   1 from canonical.launchpad.webapp.interfaces import (
   2     IStoreSelector, MAIN_STORE, AUTH_STORE,
   3     MASTER_FLAVOR, SLAVE_FLAVOR)
   4 
   5 master_store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
   6 
   7 master_obj = store.find(Person, Person.name == 'zeca')
   8 
   9 slave_store = getUtility(IStoreSelector).get(MAIN_STORE, SLAVE_FLAVOR)
  10 
  11 slave_obj = store.find(Person, Person.name == 'zeca')

If you don't need to update, but require up-to-date data, you should use the default flavor. (eg. most views - the object you are viewing might just have been created). This will retrieve the master unless the load balancer is sure all changes made by the current client have propagated to the replica databases.

   1 from canonical.launchpad.webapp.interfaces import (
   2     IStoreSelector, MAIN_STORE, AUTH_STORE, DEFAULT_FLAVOR)
   3 
   4 store = getUtility(IStoreSelector).get(MAIN_STORE, DEFAULT_FLAVOR)
   5 
   6 result = store.find(Person, Person.name == 'zeca')

See more details about this in https://lists.ubuntu.com/mailman/private/launchpad/2008-August/031418.html

Security, authentication

How can I do get the current user in a database class?

How can I protect a method based on one of its parameter?

Email Notifications

When I need to send a notification for a person/team, how do I know what email address(es) I have to send the notification to?

Web UI

When should I use SQLObjectEditView and SQLObjectAddView

How do I use the General Form

How do I perform an action after an autogenerated edit form has been successfully submitted?

How do I perform an action after an autogenerated add form has been successfully submitted?

How can I redirect a user to a new object just created from an autogenerated add form?

How do I format dates and times in page templates?

How should I generate notices like "Added Bug #1234" to the top of the page?

Launchpad API

How do I add a new celebrity?

Global Configuration

How do I add items to launchpad.conf?

key_name must be a valid Python identifier.

section_name must be a valid Python identifier.

How are these default values changed for specific environments?

The default configuration values are overridden in /configs/<environment>/launchpad-lazr.conf. Notable environments include:

The syntax for overriding configuration values is the same as the syntax for defining them.

How do I use the items listed in launchpad.conf?

How can I temporarily override configuration variables in tests?

Since configuration is specified by a global object, canonical.config.config, tests for configuration-dependent will need to temporarily change configuration values.

Here's how to do it:

Note that config.push will alter the global configuration of the current environment. If you invoke it, you must call config.pop. You can make sure this happens by:

XXX: JonathanLange 2008-09-17: pushConfig in codeimport.tests.test_dispatcher should be moved into TestCase and this FAQ updated accordingly.

Testing

What kind of tests should we use?

How do I run just one doctest file, e.g. `lib/canonical/launchpad/doc/mytest.txt`?

What about running just one pagetest story, e.g. `lib/canonical/launchpad/pagetests/initial-bug-contacts`?

What about running a standalone pagetest, e.g. `xx-bug-index.txt`

And if I want to execute all test except one?

How can I examine my test output with PDB?

Where can I get help on running tests?

How can I check test coverage?

How can I run only the tests for the page test layer?

Where should I put my tests: in a `test_foo.py` module, or a `foo.txt` doctest file?

How to I setup my tests namespace so I can remove unwanted import statements and other noise?

Why is my page test failing mysteriously?

I'm writing a pagetest in the standalone directory that changes some objects. Will my changes be visible to other pagetests in this directory?

Why is not my page test failing when adding extra whitespace chars inside a textarea?

What does the number in square brackets after a doctest name mean?

How do I find a backtrace?

Should I rely on the order of os.listdir?

How do I find the tests that cover the code I've changed?

Although you've written the tests first before changing code, a lot of code is also exercised by many other tests and it's not immediately obvious which those might be without running the whole test suite. There is an experimental tool to help with this at TestsFromChanges.

Sample Data

Sampledata should be innocuous, such that if it is viewed by people outside Canonical, it will not be embarrassing or reveal company-confidential information.

Where can I find a list of the sample users?

How to I make changes to the sample Launchpad database?

If you are making a schema change, please see this page instead.

You shouldn't make changes to sample data unless you are submitting your changes to db-devel. You can save the state of the development site's database and replace the current-dev.sql so that everyone can see a working site.

Note that as a side effect of the sampledata being automatically generated, you will often get difficult to resolve conflicts if you have modified the sample data and attempt to merge in another branch that has also modified it. To work around this, it is recommended that your sampledata changes are always maintained as list of statements in a .sql file so that you can easily reset your current.sql to the launchpad trunk and replay your changes against it (psql -d launchpad_dev -f mysampledatachanges.sql).

I've only made minor changes to the sample data, but the diff is huge!

Code Reviews

How do I use Meld with Bazaar to do code reviews?

Bazaar

How do I revert some revisions from my archive?

I merged from another branch and did a lot of changes without first committing the merge. How can I undo only the merge?

If you are sure your changes don't touch the same code as the merge does, you can do this and it'll probably work:

On the other hand, if your changes touch the same code as the merge, it's better to do the following

How can I get error failures from PQM by email?

How do I set up connection multiplexing for my SSH session?

Rollouts

See ReleaseCycles for more information.

When will my code changes end up on production?

I have an urgent bug fix

What is the staging server?

Changing the launchpad dependency debs

The launchpad deb dependencies are in the ~launchpad PPA: https://launchpad.net/~launchpad/+archive/ppa. To change them, follow the instructions on the Launchpad PPA page.

How do I run scripts with the Python environment of Launchpad?

Use bin/py in your trunk, or whatever other branch you want to use. You might also want to use bin/ipy, if you like IPython.

Buildout

We use zc.buildout for our build system.

Where can I read about it?

You can read about how we set it up in doc/buildout.txt in your trunk. This includes instructions for adding and upgrading distributions.

You can read more general information at the buildout site, buildout.org.

How can I find out what we are using via buildout?

For direct dependencies, look in setup.py (in the top directory of Launchpad). If it is in the "install_requires" argument, then we are getting it from buildout.

For direct and indirect dependencies, look in bin/run (many scripts will do, but this one is how we start Launchpad). The sys.path will show you everything that we get from eggs (that is, via buildout).

How can I find out what we are using via sourcecode?

The canonical answer is to recursively find all symlinks in lib/, bzrplugins/, and optionalbzrplugins/. Here's a find incantation that will list the results: find lib bzrplugins optionalbzrplugins -lname '*/sourcecode/*' -ls .

You should also be able to look at utilities/sourcedeps.conf. This controls what is updated when sources are updated (for instance, via rocketfuel-get). However, it is maintained manually, so it could be behind the times if someone makes a mistake.

You do NOT answer this question by looking in the sourcecode directory. The sourcecode directory is typically a shared resource for all your branches, so it does not necessarily reflect what the current branch is using. It is also not cleaned out by any of our scripts.

Is the ultimate goal to completely get rid of sourcecode, so that all packages come from buildout?

That was our original goal. We still want to shrink it to a very small size, maybe to exclusively hold canonical-identify-provider and shipit. We then might connect those in using develop eggs, rather than symlinks in lib/.

How do we make custom distributions?

See doc/buildout.txt, mentioned above. Also see this proposed policy.

Working with our open-source components (lazr.*)

See HackingLazrLibraries.

Where to go for other help

If you have encountered a problem, maybe someone else has and has already documented it on the SolutionsLog. Go have a look! If have a problem and you find a solution, document it there!

Unanswered questions