Diff for "Translations/GenerateTemplatesOnTestServers"

Not logged in - Log In / Register
Differences between revisions 27 and 42 (spanning 15 versions)
⇤ ← Revision 27 as of 2010-04-26 14:07:52 →
Size: 5091
Editor: jtv
Comment:
← Revision 42 as of 2020-09-17 12:48:23 → ⇥
Size: 5059
Editor: cjwatson
Comment: update dogfood instructions for modern TTBs
Deletions are marked like this. Additions are marked like this.
Line 17: Line 17:
We currently [[Bug:569108|fail while cleaning up the job]] from the database.
Line 26: Line 24:
=== Firewall ===

The slaves in the `dogfood` build farm must be allowed to make `http` connections to whatever codehosting server you will be using.

'''TODO: Ensure this—[[Bug:499407|bug 499407]].'''


=== Config ===

Make sure that generating templates is enabled in the Launchpad lazr config.
 * Enabled for `dogfood`.
 * Enabled on development systems.
 * Enabled for `staging`.

The configuration item in question is `generate_templates` in the `[rosetta]` section.
Line 46: Line 28:
We don't yet know for sure what real-world branches will work right out of the box. We must fix that! All recently generated templates on the production server will be on [[https://translations.launchpad.net/~vcs-imports/+imports|the import queue for vcs-imports]], so that's the best place to look for fully pottery-compatible branches.
Line 58: Line 40:
'''TODO: Find a good real-world sample branch for testing—preferably small but with multiple templates.'''
Line 61: Line 41:
 * {{{gimp}}}, but it's large.  * {{{lp:gedit}}}: used to work with pottery (and no pots in the branch), but as of 2010-08-27 it doesn't seem to work.
 * {{{lp:gimp}}}, but it's large.
Line 64: Line 45:
 * {{{lp:gconf}}}: OK, one template  * {{{lp:gconf}}}: OK, one template, no pots in the branch.
Line 72: Line 53:
There must be at least one ''virtualized'' 386 builder active.

With a bit of hacking, Julian can get a job to run on a non-virtualized builder. This makes the tail of the build log visible on the builder page.		 There must be at least one suitable amd64 builder active. "Suitable" would be a virtualized builder for production runs, but here we set our jobs up for running on nonvirtualized ones so we can get at the logs. (It'll also make the tail of the log visible on the builder page).
Line 77: Line 56:
== Procedure == == Testing: Creating jobs ==
Line 79: Line 58:
Set up a product series on both `staging` and `dogfood`. On `staging` we can test the generation of jobs, though we can't run them there.
Line 81: Line 60:
Set up a pottery-compatible branch on `staging`; set it as the product series' development branch. Do all this on `staging`:
Line 83: Line 62:
''Register'' a branch on `dogfood`, but do not push it (as `dogfood` has no codehosting). Set it as the product series' development branch.  * Set up a productseries.
 * Set up a pottery-compatible branch, e.g. by copying one off production.
 * Set the branch as the productseries' development branch.
 * Enable template imports for the productseries.
 * Push a change to the branch.
Line 85: Line 68:
Configure translation template imports for both product series.

Push a change to your branch on staging. Wait for it to be scanned.

Once the job is scanned, a triplet of matching records should appear in the staging database: a {{{BranchJob}}}, a {{{Job}}}, and a {{{BuildQueue}}}. They are matched on {{{BranchJob.job}}}, {{{Job.id}}}, and {{{BuildQueue.job}}}. In SQL it should be at the top of...		 Once your branch change is scanned, a triplet of matching records should appear in the `staging` database: a {{{BranchJob}}}, a {{{Job}}}, and a {{{BuildQueue}}}. They are matched on {{{BranchJob.job}}}, {{{Job.id}}}, and {{{BuildQueue.job}}}. In SQL it should be at the top of...
Line 97: Line 76:
    BuildQueue.job_type = 4 AND
    Job.status = 0		     BuildQueue.job_type = 4
Line 102: Line 80:
Copy what you find directly into the `dogfood` database. Start with the `Job`: It probably has `status` set to 0 for "not started yet."


== Testing: Running jobs ==

We can create job records directly in the `dogfood` database, which can't generate the jobs itself but which can dispatch them to its build farm.

Do all this on `dogfood`:

 * Set up a productseries.
 * ''Register'' a branch on `dogfood`, but ''do not push it'' (as `dogfood` has no codehosting).
 * Set the branch as the productseries' development branch.
 * Enable template imports for the productseries.

Now we create the job records. Run:
Line 104: Line 96:
INSERT INTO Job (status) values (0) RETURNING id; make -C /srv/launchpad.net/codelines/current iharness
Line 107: Line 99:
Note the id that this returns. Now, create a `BranchJob` using that job id, the id of your branch ''on `dogfood`,'' and a copy of the `json_data` as created on `staging`: And in that, run:
Line 109: Line 101:
INSERT INTO BranchJob (job, branch, job_type, json_data)
VALUES (<jobid>, <branchid>, 6, <json>);		 >>> import transaction
>>> from lp.code.interfaces.branchlookup import IBranchLookup
>>> from lp.translations.interfaces.translationtemplatesbuild import ITranslationTemplatesBuildSource
>>> branch, _ = getUtility(IBranchLookup).getByLPPath('gconf') # or some other branch path
>>> _ = getUtility(ITranslationTemplatesBuildSource).create(branch).queueBuild()
>>> transaction.commit()
Line 113: Line 109:
Finally, the `BuildQueue` entry:
{{{
INSERT INTO BuildQueue (job, job_type)
VALUES (<jobid>, 4);
}}}

This should be enough to get this job onto the build farm. Keep an eye on https://dogfood.launchpad.net/builders where it should appear shortly (and then, after processing, disappear again).

Once the job is successfully completed, any generated templates should appear on the productseries' translations import queue on `dogfood`.		 Keep an eye on https://dogfood.paddev.net/builders where it should appear shortly (and then, after processing, disappear again). Also, if everything works, templates should appear on your productseries' import queue!

Trying out template generation on dogfood/staging

Right now we have no good test platform for this: staging has no build farm, and dogfood has no codehosting. Expect this page to change a lot as we work and learn. We already have pages about doing this on a development system that may also help.

Internal Soyuz documentation on dealing with dogfood.

Approach

For now, we'll have to live with a split test setup:

  • Push changes to generate jobs on staging.

  • Copy the jobs over to the dogfood database.

  • Let the dogfood build farm generate the templates.

Known Problems

intltool recognition

It looks like gettext has changed, and our check for intltool branches now also accepts some plain gettext branches. This happens with lp:gawk, which pottery mistakenly recognizes as intltool but doesn't process successfully because there is no GETTEXT_PACKAGE definition.

Checklist

Suitable branches

Make sure you have a branch that is pottery-compatible. (We call the code that generates templates from code "pottery"). For testing, it should normally not contain any .pot files. Otherwise we'd just end up importing those instead of ones we generated.

All recently generated templates on the production server will be on the import queue for vcs-imports, so that's the best place to look for fully pottery-compatible branches.

According to wgrant, on many branches, editing configure.ac to set GETTEXT_PACKAGE does the trick. See bug 568355.

You can easily check if a branch will work by generating a template locally. Run the following command from the LP tree in the packages root directory: 

scripts/rosetta/pottery-generate-intltool.py

It outputs the template files it creates. It uses the same code as the build slave.

Candidate branches:

  • lp:gedit: used to work with pottery (and no pots in the branch), but as of 2010-08-27 it doesn't seem to work.

  • lp:gimp, but it's large.

  • lp:ubuntu/gawk does not use intltool, but pottery seems to think that it does. See bug 568372.

  • lp:libgnomeui: OK, one template

  • lp:gconf: OK, one template, no pots in the branch.

Once you have a suitable branch, you can copy it to staging's codehosting using "bzr push -d lp:<branch> lp://staging/<branch> --use-existing-dir"

Builders

There must be at least one suitable amd64 builder active. "Suitable" would be a virtualized builder for production runs, but here we set our jobs up for running on nonvirtualized ones so we can get at the logs. (It'll also make the tail of the log visible on the builder page).

Testing: Creating jobs

On staging we can test the generation of jobs, though we can't run them there.

Do all this on staging:

  • Set up a productseries.
  • Set up a pottery-compatible branch, e.g. by copying one off production.
  • Set the branch as the productseries' development branch.
  • Enable template imports for the productseries.
  • Push a change to the branch.

Once your branch change is scanned, a triplet of matching records should appear in the staging database: a BranchJob, a Job, and a BuildQueue. They are matched on BranchJob.job, Job.id, and BuildQueue.job. In SQL it should be at the top of... 

SELECT *
FROM BranchJob
JOIN Job ON Job.id = BranchJob.job
JOIN BuildQueue ON BuildQueue.job = Job.id
WHERE
    BranchJob.job_type = 6 AND
    BuildQueue.job_type = 4
ORDER BY Job.id DESC;

It probably has status set to 0 for "not started yet."

Testing: Running jobs

We can create job records directly in the dogfood database, which can't generate the jobs itself but which can dispatch them to its build farm.

Do all this on dogfood:

  • Set up a productseries.

  • Register a branch on dogfood, but do not push it (as dogfood has no codehosting).

  • Set the branch as the productseries' development branch.
  • Enable template imports for the productseries.

Now we create the job records. Run: 

make -C /srv/launchpad.net/codelines/current iharness

And in that, run: 

>>> import transaction
>>> from lp.code.interfaces.branchlookup import IBranchLookup
>>> from lp.translations.interfaces.translationtemplatesbuild import ITranslationTemplatesBuildSource
>>> branch, _ = getUtility(IBranchLookup).getByLPPath('gconf')  # or some other branch path
>>> _ = getUtility(ITranslationTemplatesBuildSource).create(branch).queueBuild()
>>> transaction.commit()

Keep an eye on https://dogfood.paddev.net/builders where it should appear shortly (and then, after processing, disappear again). Also, if everything works, templates should appear on your productseries' import queue!

Translations/GenerateTemplatesOnTestServers (last edited 2020-09-17 12:48:23 by cjwatson)