Trying out template generation on dogfood/staging

Right now we have no good test platform for this: staging has no build farm, and dogfood has no codehosting. Expect this page to change a lot as we work and learn. We already have pages about doing this on a development system that may also help.

Internal Soyuz documentation on dealing with dogfood.

Approach

For now, we'll have to live with a split test setup:

• Push changes to generate jobs on staging.

• Copy the jobs over to the dogfood database.

• Let the dogfood build farm generate the templates.

Known Problems

We currently fail while cleaning up the job from the database.

When re-running on a non-virtualized slave (so we can get to the logs and filesystem), the builder log gets stuck at the chroot unpack stage, even though the slave log says it continues to do its job.

intltool recognition

It looks like gettext has changed, and our check for intltool branches now also accepts some plain gettext branches. This happens with lp:gawk, which pottery mistakenly recognizes as intltool but doesn't process successfully because there is no GETTEXT_PACKAGE definition.

Checklist

Suitable branches

Make sure you have a branch that is pottery-compatible. (We call the code that generates templates from code "pottery"). For testing, it should normally not contain any .pot files. Otherwise we'd just end up importing those instead of ones we generated.

We don't yet know for sure what real-world branches will work right out of the box. We must fix that!

According to wgrant, on many branches, editing configure.ac to set GETTEXT_PACKAGE does the trick. See bug 568355.

You can easily check if a branch will work by generating a template locally. Run the following command from the LP tree in the packages root directory:

scripts/rosetta/pottery-generate-intltool.py

It outputs the template files it creates. It uses the same code as the build slave.

Candidate branches:

• lp:gedit: works with pottery, no pots in the branch.

• lp:gimp, but it's large.

• lp:ubuntu/gawk does not use intltool, but pottery seems to think that it does. See bug 568372.

• lp:libgnomeui: OK, one template

• lp:gconf: OK, one template

Once you have a suitable branch, you can copy it to staging's codehosting using "bzr push -d lp:<branch> lp://staging/<branch> --use-existing-dir"

Builders

There must be at least one suitable 386 builder active. "Suitable" would be a virtualized builder for production runs, but here we set our jobs up for running on nonvirtualized ones so we can get at the logs. (It'll also make the tail of the log visible on the builder page).

Testing: Creating jobs

On staging we can test the generation of jobs, though we can't run them there.

Do all this on staging:

• Set up a productseries.
• Set up a pottery-compatible branch, e.g. by copying one off production.
• Set the branch as the productseries' development branch.
• Enable template imports for the productseries.
• Push a change to the branch.

Once your branch change is scanned, a triplet of matching records should appear in the staging database: a BranchJob, a Job, and a BuildQueue. They are matched on BranchJob.job, Job.id, and BuildQueue.job. In SQL it should be at the top of...

SELECT *
FROM BranchJob
JOIN Job ON Job.id = BranchJob.job
JOIN BuildQueue ON BuildQueue.job = Job.id
WHERE
    BranchJob.job_type = 6 AND
    BuildQueue.job_type = 4
ORDER BY Job.id DESC;

It probably has status set to 0 for "not started yet."

Testing: Running jobs

We can create job records directly in the dogfood database, which can't generate the jobs itself but which can dispatch them to its build farm.

Do all this on dogfood:

• Set up a productseries.

• Register a branch on dogfood, but do not push it (as dogfood has no codehosting).

• Set the branch as the productseries' development branch.
• Dive into the database and figure out the id for the branch.
• Enable template imports for the productseries.

Now we manually create the job records (more or less as automatically created in the staging database) in the dogfood database, but using locally valid ids of course.

Start with the Job, which really has nothing interesting except the id:

INSERT INTO Job (status) values (0) RETURNING id;

Note the id that this returns. Now, create a BranchJob using that job id, the id of your branch on dogfood, and a copy of the json_data as created on staging. The json_data looks like:  {'branch_url': 'http://bazaar.staging.launchpad.net/~my/project/branch'}  (substitute your branch URL).

INSERT INTO BranchJob (job, branch, job_type, json_data)
VALUES (<jobid>, <branchid>, 6, <json>);

Finally, create the BuildQueue entry. Normally we'd leave virtualized to null, which would give us a virtualized builder, but for Q/A it's better to use a nonvirtualized one such as ferraz. We can get logs from those slaves, which isn't possible with the virtualized builder we also have.

INSERT INTO BuildQueue (job, job_type, virtualized)
VALUES (<jobid>, 4, false);

This should be enough to get this job onto the build farm. Keep an eye on https://dogfood.launchpad.net/builders where it should appear shortly (and then, after processing, disappear again). Also, if everything works, templates should appear on your productseries' import queue!