Size: 2704
Comment:
|
Size: 5475
Comment:
|
Deletions are marked like this. | Additions are marked like this. |
Line 4: | Line 4: |
[[https://wiki.canonical.com/Launchpad/Soyuz/CommonTasks|Internal Soyuz documentation on dealing with dogfood.]] |
|
Line 12: | Line 15: |
== Known Problems == === intltool recognition === It looks like gettext has changed, and our check for intltool branches now also accepts some plain gettext branches. This happens with {{{lp:gawk}}}, which pottery mistakenly recognizes as intltool but doesn't process successfully because there is no {{{GETTEXT_PACKAGE}}} definition. |
|
Line 14: | Line 23: |
=== Config === Make sure that generating templates is enabled in the Launchpad lazr config. * Enabled for `dogfood`. * Enabled on development systems. * Being enabled for `staging`. '''TODO: what production-configs revision? How will we know it's landed?''' The configuration item in question is `generate_templates` in the `[rosetta]` section. |
|
Line 31: | Line 30: |
According to `wgrant`, editing `configure.ac` to set `GETTEXT_PACKAGE` ("or something like that") does the trick. | According to `wgrant`, on many branches, editing `configure.ac` to set `GETTEXT_PACKAGE` does the trick. See [[Bug:568355|bug 568355]]. |
Line 33: | Line 32: |
'''TODO: Easy litmus test to see if a branch will work with pottery.''' | You can easily check if a branch will work by generating a template locally. Run the following command from the LP tree in the packages root directory: |
Line 35: | Line 34: |
'''TODO: Find good real-world sample branch for testing—preferably small but with multiple templates.''' | {{{ scripts/rosetta/pottery-generate-intltool.py }}} It outputs the template files it creates. It uses the same code as the build slave. Candidate branches: * {{{lp:gedit}}}: works with pottery, no pots in the branch. * {{{lp:gimp}}}, but it's large. * {{{lp:ubuntu/gawk}}} does not use intltool, but pottery seems to think that it does. See [[Bug:568372|bug 568372]]. * {{{lp:libgnomeui}}}: OK, one template * {{{lp:gconf}}}: OK, one template |
Line 40: | Line 51: |
== Procedure == | === Builders === |
Line 42: | Line 53: |
Set up a product series on both `staging` and `dogfood`. | There must be at least one suitable 386 builder active. "Suitable" would be a virtualized builder for production runs, but here we set our jobs up for running on nonvirtualized ones so we can get at the logs. (It'll also make the tail of the log visible on the builder page). |
Line 44: | Line 55: |
Set up a pottery-compatible branch on `staging`; set it as the product series' development branch. | |
Line 46: | Line 56: |
''Register'' a branch on `dogfood`, but do not push it (as `dogfood` has no codehosting). Set it as the product series' development branch. | == Testing: Creating jobs == |
Line 48: | Line 58: |
Configure translation template imports for both product series. | On `staging` we can test the generation of jobs, though we can't run them there. |
Line 50: | Line 60: |
Push a change to your branch on staging. Wait for it to be scanned. | Do all this on `staging`: |
Line 52: | Line 62: |
Once the job is scanned, a triplet of matching records should appear in the staging database: a {{{BranchJob}}}, a {{{Job}}}, and a {{{BuildQueue}}}. They are matched on {{{BranchJob.job}}}, {{{Job.id}}}, and {{{BuildQueue.job}}}. In SQL: | * Set up a productseries. * Set up a pottery-compatible branch, e.g. by copying one off production. * Set the branch as the productseries' development branch. * Enable template imports for the productseries. * Push a change to the branch. Once your branch change is scanned, a triplet of matching records should appear in the `staging` database: a {{{BranchJob}}}, a {{{Job}}}, and a {{{BuildQueue}}}. They are matched on {{{BranchJob.job}}}, {{{Job.id}}}, and {{{BuildQueue.job}}}. In SQL it should be at the top of... |
Line 58: | Line 74: |
WHERE BranchJob.job_type = 6 ORDER BY Job.id |
WHERE BranchJob.job_type = 6 AND BuildQueue.job_type = 4 ORDER BY Job.id DESC; |
Line 61: | Line 79: |
It probably has `status` set to 0 for "not started yet." == Testing: Running jobs == We can create job records directly in the `dogfood` database, which can't generate the jobs itself but which can dispatch them to its build farm. Do all this on `dogfood`: * Set up a productseries. * ''Register'' a branch on `dogfood`, but ''do not push it'' (as `dogfood` has no codehosting). * Set the branch as the productseries' development branch. * Enable template imports for the productseries. Now we directly create the job records (more or less as automatically created in the `staging` database) in the `dogfood` database, but using locally valid ids of course. To do this, run: {{{ /home/launchpad/bin/make-translation-templates-job <project name> <productseries name> <http branch url> }}} Here, * {{{project name}}} is the identifying name of your product on {{{dogfood}}}. * {{{productseries name}}} is the identifying name of your productseries in that product, e.g. "trunk" * {{{http branch url}}} is the public URL, using the {{{http}}} protocol, of your real branch on production or staging. For example, {{{http://bazaar.launchpad.net/~vcs-imports/gconf/trunk}}} This will create a job to be run on a ''nonvirtualized'' builder. If you want to change this, set the {{{VIRTUALIZED}}} variable to "true" (case-insensitive): {{{ env VIRTUALIZED=TRUE /home/launchpad/bin/make-translation-templates-job <project name> <productseries name> <http branch url> }}} A lousy interface but it saved some time. Do not worry about what identity you're running under; the script automatically operates as {{{launchpad}}}. Keep an eye on https://dogfood.launchpad.net/builders where it should appear shortly (and then, after processing, disappear again). Also, if everything works, templates should appear on your productseries' import queue! |
Trying out template generation on dogfood/staging
Right now we have no good test platform for this: staging has no build farm, and dogfood has no codehosting. Expect this page to change a lot as we work and learn. We already have pages about doing this on a development system that may also help.
Internal Soyuz documentation on dealing with dogfood.
Approach
For now, we'll have to live with a split test setup:
Push changes to generate jobs on staging.
Copy the jobs over to the dogfood database.
Let the dogfood build farm generate the templates.
Known Problems
intltool recognition
It looks like gettext has changed, and our check for intltool branches now also accepts some plain gettext branches. This happens with lp:gawk, which pottery mistakenly recognizes as intltool but doesn't process successfully because there is no GETTEXT_PACKAGE definition.
Checklist
Suitable branches
Make sure you have a branch that is pottery-compatible. (We call the code that generates templates from code "pottery"). For testing, it should normally not contain any .pot files. Otherwise we'd just end up importing those instead of ones we generated.
We don't yet know for sure what real-world branches will work right out of the box. We must fix that!
According to wgrant, on many branches, editing configure.ac to set GETTEXT_PACKAGE does the trick. See bug 568355.
You can easily check if a branch will work by generating a template locally. Run the following command from the LP tree in the packages root directory:
scripts/rosetta/pottery-generate-intltool.py
It outputs the template files it creates. It uses the same code as the build slave.
Candidate branches:
lp:gedit: works with pottery, no pots in the branch.
lp:gimp, but it's large.
lp:ubuntu/gawk does not use intltool, but pottery seems to think that it does. See bug 568372.
lp:libgnomeui: OK, one template
lp:gconf: OK, one template
Once you have a suitable branch, you can copy it to staging's codehosting using "bzr push -d lp:<branch> lp://staging/<branch> --use-existing-dir"
Builders
There must be at least one suitable 386 builder active. "Suitable" would be a virtualized builder for production runs, but here we set our jobs up for running on nonvirtualized ones so we can get at the logs. (It'll also make the tail of the log visible on the builder page).
Testing: Creating jobs
On staging we can test the generation of jobs, though we can't run them there.
Do all this on staging:
- Set up a productseries.
- Set up a pottery-compatible branch, e.g. by copying one off production.
- Set the branch as the productseries' development branch.
- Enable template imports for the productseries.
- Push a change to the branch.
Once your branch change is scanned, a triplet of matching records should appear in the staging database: a BranchJob, a Job, and a BuildQueue. They are matched on BranchJob.job, Job.id, and BuildQueue.job. In SQL it should be at the top of...
SELECT * FROM BranchJob JOIN Job ON Job.id = BranchJob.job JOIN BuildQueue ON BuildQueue.job = Job.id WHERE BranchJob.job_type = 6 AND BuildQueue.job_type = 4 ORDER BY Job.id DESC;
It probably has status set to 0 for "not started yet."
Testing: Running jobs
We can create job records directly in the dogfood database, which can't generate the jobs itself but which can dispatch them to its build farm.
Do all this on dogfood:
- Set up a productseries.
Register a branch on dogfood, but do not push it (as dogfood has no codehosting).
- Set the branch as the productseries' development branch.
- Enable template imports for the productseries.
Now we directly create the job records (more or less as automatically created in the staging database) in the dogfood database, but using locally valid ids of course.
To do this, run:
/home/launchpad/bin/make-translation-templates-job <project name> <productseries name> <http branch url>
Here,
project name is the identifying name of your product on dogfood.
productseries name is the identifying name of your productseries in that product, e.g. "trunk"
http branch url is the public URL, using the http protocol, of your real branch on production or staging. For example, http://bazaar.launchpad.net/~vcs-imports/gconf/trunk
This will create a job to be run on a nonvirtualized builder. If you want to change this, set the VIRTUALIZED variable to "true" (case-insensitive):
env VIRTUALIZED=TRUE /home/launchpad/bin/make-translation-templates-job <project name> <productseries name> <http branch url>
A lousy interface but it saved some time. Do not worry about what identity you're running under; the script automatically operates as launchpad.
Keep an eye on https://dogfood.launchpad.net/builders where it should appear shortly (and then, after processing, disappear again). Also, if everything works, templates should appear on your productseries' import queue!