Diff for "LoggingOopses"

Not logged in - Log In / Register

Differences between revisions 17 and 18
Revision 17 as of 2009-10-29 07:55:49
Size: 5999
Editor: jtv
Comment:
Revision 18 as of 2009-10-29 13:40:38
Size: 6822
Editor: jtv
Comment:
Deletions are marked like this. Additions are marked like this.
Line 64: Line 64:
'''TODO: Can map multiple scripts' oopses to one error_dir.''' '''TODO: Multiple scripts' oopses can map to one error_dir.'''
Line 96: Line 96:
This is not a regular {{{launchpad}}} branch but once from a separate project, so create your local branch somewhere ''outside'' the directory (usually called {{{lp-branches}}}) where you keep your regular Launchpad branches!  So: This is not a regular {{{launchpad}}} branch but once from a separate project, so you'll want to create your local branch ''outside'' the directory (usually called {{{lp-branches}}}) where you keep your regular Launchpad branches!
Line 98: Line 98:

==== Creating a repository ====

I'd suggest setting up a new repository next to the one that holds your Launchpad branches:
Line 103: Line 107:
mkdir -p lp-production-configs bzr init-repo lp-production-configs
Line 106: Line 110:
}}}
Line 107: Line 112:
# Create your working branch. You'll have to add this to your {{{.bazaar/locations.conf}}} to make use of this:
{{{
[/home/me/canonical/lp-production-configs]
pqm_email = launchpad@pqm.canonical.com
submit_branch = bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/lp-production-configs/trunk
public_branch = bzr+ssh://bazaar.launchpad.net/~me/lp-production-configs
public_branch:policy = appendpath
push_location = bzr+ssh://me@bazaar.launchpad.net/%7Eme/lp-production-configs
push_location:policy = appendpath
}}}


==== Branching lp-production-configs ====

You'll need to branch off {{{lp:~launchpad-pqm/lp-production-configs/trunk}}}. If you set up the repository as suggested above:
{{{
cd ~/canonical/lp-production-configs

Logging Oopses

WARNING: Work in progress. Needs improvement. If you think you know better, you probably do—so hit Edit!

This page describes how to add oops logging to your script. All fatal errors that need operator or developer attention should be logged as oopses.

Assumptions for examples:

  • Your Launchpad login name is "me."
  • The script you're working on is called frobnicate.

  • Your oopses have a prefix code of FROB.
  • Adding the oops logging is fixing bug 999999.
  • You're working in a branch of devel or db-devel called bug-999999.

Where you see these in the examples, replace them with whatever you've got.

Choosing an oops code

TODO: How exactly are oops codes chosen?

The first thing you do is pick a code for your oopses. The code is included in the file names for oops reports, and will be the first indication that an oops was generated by your script.

Oops codes should be:

  • All upper-case ASCII letters: [A-Z]+

  • Short, typically 2—4 letters.
  • Unique.

Code

The oops logging machinery comes from Zope, and is lightly extended in Lazr. It was designed for web applications, so expect some twists and turns as you try to squeeze a script error into the mould of a web request.

TODO: Verify that bit of possible nonsense.

TODO: Describe what's needed in the code.

Configuration

In your branch

You'll need to add some configuration items to the configuration files in configs/. This is in the bug-999999 branch.

TODO: Refer to configs documentation.

Your script must have a configuration section in configs/schema-lazr.conf. It should probably look a lot like this:

[frobnicate]
dbuser: frobnicate
storm_cache: generational
storm_cache_size: 500

If the section does not exist yet, this snippet should be a good start for creating it.

To this section, add blank default config items for your oopses:

oops_prefix: none
error_dir: none
copy_to_zlog: false

TODO: Explain what these do.

TODO: Multiple scripts' oopses can map to one error_dir.

The specific values will go into specific configs for various setups. Here, if the configuration files do not have frobnicate sections, just create them by adding the header:

[frobnicate]

In configs/development/launchpad-lazr.conf add these to the frobnicate section:

oops_prefix: FROB
error_dir: /var/tmp/frobnicate.test

Note the "T" prefix to the oops code, and the ".test" suffix to the oops directory.

Similarly, in configs/testrunner/launchpad-lazr.conf, under in the frobnicate header, add:

oops_prefix: TFROB

The testrunner configuration is derived from the development one, so no need to specify the same error_dir again here.

Make sure it works

Once you've configured and programmed your oopses, you can run your tests to trigger them and you will see the oops reports appearing in /var/tmp/frobnicate.test/, neatly categorized by date. Since they're in /var/tmp/, they'll be cleaned up on reboot.

In the production world

You need to add configuration items to the production configs as well. Make your changes in a branch of lp-production-configs, separate from the bug-999999 branch.

This is not a regular launchpad branch but once from a separate project, so you'll want to create your local branch outside the directory (usually called lp-branches) where you keep your regular Launchpad branches!

Creating a repository

I'd suggest setting up a new repository next to the one that holds your Launchpad branches:

cd ~/canonical
# (Or wherever, as long as it isn't where your Launchpad branches are.)

# Set up a place for lp-production-configs branches.
bzr init-repo lp-production-configs
cd lp-production-configs
bzr branch lp:~launchpad-pqm/lp-production-configs/trunk

You'll have to add this to your .bazaar/locations.conf to make use of this:

[/home/me/canonical/lp-production-configs]
pqm_email = launchpad@pqm.canonical.com
submit_branch = bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/lp-production-configs/trunk
public_branch = bzr+ssh://bazaar.launchpad.net/~me/lp-production-configs
public_branch:policy = appendpath
push_location = bzr+ssh://me@bazaar.launchpad.net/%7Eme/lp-production-configs
push_location:policy = appendpath

Branching lp-production-configs

You'll need to branch off lp:~launchpad-pqm/lp-production-configs/trunk. If you set up the repository as suggested above:

cd ~/canonical/lp-production-configs
bzr branch trunk production-configs-bug-999999
cd production-configs-bug-999999

TODO: Make the setup use the appropriate LP setup variables so it's no longer based on my local directory structures.

TODO: bzr setup to make the push locations etc. automatic.

Now make your changes in production-configs-bug-999999. Add the following snippet to staging-lazr.conf, in the config section for your script. If the section did not exist, create an empty one.

oops_prefix: SFROB
error_dir: /srv/staging.launchpad.net/staging-logs/frobnicate

TODO: A different hostname may be appropriate.

To production/launchpad-lazr.conf, add this to the frobnicate section:

oops_prefix: FROB
error_dir: /srv/launchpad.net/production-logs/frobnicate

Commit the changes and push your branch:

bzr push lp:~me/lp-production-configs/production-configs-bug-999999

Now propose this branch for merging into lp:~launchpad-pqm/lp-production-configs. This is the default target for the merge proposal. The branch needs to be reviewed by one of the LOSA's ("canonical-losas"); this is also the default choice of reviewer.

Make sure it works

Before you go to PQM, have the change cowboy-patched onto staging for testing. Then produce the oops there.

After a while, the oops report you generated should be mirrored to one of the directories in devpad:/srv/launchpad.net-logs/staging/. It's typically the subdirectory with the same name as the server the script ran on, in another subdirectory with today's date.

At the time of writing, the oopses from staging are synced to devpad every 3 minutes.

Land

Once you're satisfied that the oops are logged as expected, submit your lp-production-configs to PQM for landing. Use the regular bzr pqm-submit for this:

bzr pqm-submit -m "[r=myreviewer][bug=999999] Configure oops logging for frobnicate."

And now... wait!

LoggingOopses (last edited 2011-10-31 10:43:45 by lifeless)