Logging Oopses

WARNING: Needs improvement. If you think you know better, you probably do—so hit Edit!

This page describes how to add oops logging to your script. All fatal errors that need operator or developer attention should be logged as oopses.

Assumptions for examples:

• Your Launchpad login name is "me."

• The script you're working on is called frobnicate.

• Your oopses have a prefix code of FROB.
• Adding the oops logging is fixing bug 999999.

• You're working in a branch of devel or db-devel called bug-999999.

Where you see these in the examples, replace them with whatever you've got.

You'll want to run this before you start with any of this, to hide the differences between local setups:

. ~/.rocketfuel-env.sh

Choosing an oops code

TODO: How exactly are oops codes chosen?

The first thing you do is pick a code for your oopses. The code is included in the file names for oops reports, and will be the first indication that an oops was generated by your script.

Oops codes should be:

• All upper-case ASCII letters: [A-Z]+

• Short, typically 2—4 letters.
• Unique.

Code

The oops logging machinery comes from Zope, and is lightly extended in Lazr. It was designed for web applications, so expect some twists and turns as you try to squeeze a script error into the mould of a web request.

In a nutshell:

import sys

# Import oops logging support.
from canonical.launchpad.webapp import errorlog

# ...

# Get traceback information.
exception_info = sys.exc_info()

# Describe the failure as a list of key/value pairs.
description = [
    ('key1', value1),
    ('key2', value2),
    ]

# Create a request to hold your failure description, as if
# the failure happened while servicing a web request.
request = errorlog.ScriptRequest(description)

# Report the oops.
errorlog.globalErrorUtility.raising(exception_info, request)

Generating oopses is hard to test, especially when it comes to proper configuration. If your script is breaking while trying to report an oops, you may not get any notice about it. So keep this part of your script simple!

In twistd / twisted daemons

from lp.services.twistedsupport.loggingsupport import set_up_oops_reporting
set_up_oops_reporting('configsection', 'loggername')

The config section needs its schema changed to have an oops prefix and errordir entries, just like any other process.

Configuration

The new type of oops needs to be configured with its own oops prefix code, as well as a storage location for its oops reports. We have separate configurations for test runs, local runs, staging, and production. It's important to get all of these right: the test suite can't catch configuration mistakes except in its own config.

In your branch

Still in the bug-999999 branch, you'll be adding items to two configuration files in configs/.

Your script must have a configuration section in configs/schema-lazr.conf. It should probably look a lot like this:

[frobnicate]
dbuser: frobnicate
storm_cache: generational
storm_cache_size: 500

If the section does not exist yet, this snippet should be a good start for creating it.

To this section, add blank default config items for your oopses:

oops_prefix: none
error_dir: none
copy_to_zlog: false

TODO: Explain what these do.

TODO: Multiple scripts' oopses can map to one error_dir.

The specific values will go into specific configs for various setups. Here, if the configuration files do not have frobnicate sections, just create them by adding the header:

[frobnicate]

In configs/development/launchpad-lazr.conf add these to the frobnicate section:

oops_prefix: FROB
error_dir: /var/tmp/frobnicate.test

Note the "T" prefix to the oops code, and the ".test" suffix to the oops directory.

Similarly, in configs/testrunner/launchpad-lazr.conf, under in the frobnicate header, add:

oops_prefix: TFROB
error_dir: /var/tmp/frobnicate

The testrunner configuration is derived from the development one, so any settings made there but not here are inherited. But keeping separate error_dir settings for these two configurations puts the oops reports from test runs and from manual local runs separate.

Make sure it works

Once you've configured and programmed your oopses, you can run your tests to trigger them and you will see the oops reports appearing in /var/tmp/frobnicate.test/, neatly categorized by date. Since they're in /var/tmp/, they'll be cleaned up on reboot.

You can also write tests to test OOPS generation explicitly. Unfortunately, the infrastructure for that is still rather primitive, but it can be done: see bug 567257 and bug 567689 for more information.

In the production world

You need to add configuration items to the production configs as well.

To do this, create a branch of the Launchpad production configs. We'll call the branch production-configs-bug-999999.

Make your changes in production-configs-bug-999999. Add the following snippet to staging-lazr.conf, in the config section for your script. If the section did not exist, create an empty one.

oops_prefix: SFROB
error_dir: /srv/staging.launchpad.net/staging-logs/frobnicate

If your script runs on another server, e.g. bazaar.staging.launchpad.net, then you may want a directory in /srv that's named after a different host.

To production/launchpad-lazr.conf, add this to the frobnicate section:

oops_prefix: FROB
error_dir: /srv/launchpad.net/production-logs/frobnicate

Commit the changes and push your branch:

bzr push lp:~me/lp-production-configs/production-configs-bug-999999

Get this branch through review & landed (see WorkingWithProductionConfigs). You'll definitely want to Q/A on staging before you land.

Make sure it works

Before you go to PQM, have the change cowboy-patched onto staging for testing. Then produce the oops there.

After a while, the oops report you generated should be mirrored to one of the directories in devpad:/srv/launchpad.net-logs/staging/. It's typically the subdirectory with the same name as the server the script ran on, in another subdirectory with today's date. At the time of writing, the oopses from staging are synced to devpad every 3 minutes.

And of course, after your branch lands & rolls out to edge, try testing there as well.

Oops reports

Once your oopses get synced to devpad, the next step is to get them included in the daily oops reports. If your oopses are for a new script that wasn't logging oopses before, there is no oops report yet that they would automatically be included in. You'll have to have a new oops report set up. It will show up daily like the rest of the oops reports, but with a different description in the subject line.

At the time of writing, the team to approach about this is Launchpad Foundations.