Logging Oopses

WARNING: Work in progress. Needs improvement. If you think you know better, you probably do—so hit Edit!

This page describes how to add oops logging to your script. All fatal errors that need operator or developer attention should be logged as oopses.

Assumptions for examples:

• The script you're working on is called frobnicate.

• Your oopses have a prefix code of FROB.
• Adding the oops logging is fixing bug 999999.

• You're working in a branch of devel or db-devel called bug-999999.

TODO: How are oops codes chosen?

Code

The oops logging machinery comes from Zope, and is lightly extended in Lazr. It was designed for web applications, so expect some twists and turns as you try to squeeze a script error into the mould of a web request.

TODO: Verify that bit of possible nonsense.

TODO: Describe what's needed in the code.

Configuration

In your branch

You'll need to add some configuration items to the configuration files in configs/. This is in the bug-999999 branch.

TODO: Refer to configs documentation.

Your script must have a configuration section in configs/schema-lazr.conf. It should probably look a lot like this:

[frobnicate]
dbuser: frobnicate
storm_cache: generational
storm_cache_size: 500

If the section does not exist yet, this snippet should be a good start for creating it.

To this section, add blank default config items for your oopses:

oops_prefix: none
error_dir: none
copy_to_zlog: false

TODO: Explain what these do.

TODO: Can map multiple scripts' oopses to one error_dir.

The specific values will go into specific configs for various setups. Here, if the configuration files do not have frobnicate sections, just create them by adding the header:

[frobnicate]

In configs/development/launchpad-lazr.conf add these to the frobnicate section:

oops_prefix: FROB
error_dir: /var/tmp/frobnicate.test

Note the "T" prefix to the oops code, and the ".test" suffix to the oops directory.

Similarly, in configs/testrunner/launchpad-lazr.conf, under in the frobnicate header, add:

oops_prefix: TFROB

(The testrunner configuration is derived from the development one, so no need to specify the same error_dir again here).

In the production world

You need to add configuration items to the production configs as well. Make your changes in a branch of lp-production-configs, separate from the bug-999999 branch:

bzr branch lp:~launchpad-pqm/lp-production-configs/trunk production-configs-bug-999999

Now make your changes in production-configs-bug-999999. Add the following snippet to staging-lazr.conf, in the config section for your script. If the section did not exist, create an empty one.

oops_prefix: SFROB
error_dir: /srv/staging.launchpad.net/staging-logs/frobnicate

TODO: A different hostname may be appropriate.

To production/launchpad-lazr.conf, add this to the frobnicate section:

oops_prefix: FROB
error_dir: /srv/launchpad.net/production-logs/frobnicate

Now propose this branch for merging and get it reviewed. Once it's approved, submit to PQM for landing.

TODO: How to submit to PQM?

TODO: What's the loadtest config for?