Diff for "Translations/GenerateTemplatesOnTestServers"

Not logged in - Log In / Register

Differences between revisions 8 and 22 (spanning 14 versions)
Revision 8 as of 2010-04-20 09:48:47
Size: 3567
Editor: jtv
Comment:
Revision 22 as of 2010-04-23 08:36:52
Size: 5322
Editor: jtv
Comment:
Deletions are marked like this. Additions are marked like this.
Line 4: Line 4:

[[https://wiki.canonical.com/Launchpad/Soyuz/CommonTasks|Internal Soyuz documentation on dealing with dogfood.]]
Line 11: Line 14:

== Known Problems ==

=== Slave firewalls ===

The `dogfood` slaves have to be allowed outgoing `http` to the staging and production codehosting servers.

=== Slave failure handling ===

Soyuz/buildmaster [[Bug:496574|doesn't deal well]] with the slave failures. When I first got this, the jobs kept being restarted ad infinitum, leaving the builder effectively unusable. This turned out to be because the version on dogfood had been altered for testing. It will go away.


=== intltool recognition ===

It looks like gettext has changed, and our check for intltool branches now also accepts some plain gettext branches. This happens with {{{lp:gawk}}}, which pottery mistakenly recognizes as intltool but doesn't process successfully because there is no {{{GETTEXT_PACKAGE}}} definition.
Line 38: Line 56:
According to `wgrant`, editing `configure.ac` to set `GETTEXT_PACKAGE` ("or something like that") does the trick. According to `wgrant`, on many branches, editing `configure.ac` to set `GETTEXT_PACKAGE` does the trick. See [[Bug:568355|bug 568355]].
Line 40: Line 58:
'''TODO: Easy litmus test to see if a branch will work with pottery.''' You can easily check if a branch will work by generating a template locally. Run the following command from the LP tree in the packages root directory:

{{{
scripts/rosetta/pottery-generate-intltool.py
}}}

It outputs the template files it creates. It uses the same code as the build slave.
Line 43: Line 67:

Candidate branches:
 * {{{gimp}}}, but it's large.
 * {{{lp:ubuntu/gawk}}} does not use intltool, but pottery seems to think that it does. See [[Bug:568372|bug 568372]].
 * {{{lp:libgnomeui}}}: OK, one template
 * {{{lp:gconf}}}: OK, one template
Line 69: Line 100:
ORDER BY Job.id DESC ORDER BY Job.id DESC;
Line 74: Line 105:
INSERT INTO Job (status) values (0) RETURNING id INSERT INTO Job (status) values (0) RETURNING id;
Line 77: Line 108:
Note the id that this returns. Now, create a `BranchJob` using that job id, the id of your branch on `dogfood`, and a copy of the `json_data` as created on `staging`: Note the id that this returns. Now, create a `BranchJob` using that job id, the id of your branch ''on `dogfood`,'' and a copy of the `json_data` as created on `staging`:
Line 80: Line 111:
VALUES (<jobid>, <branchid>, 6, <json>) VALUES (<jobid>, <branchid>, 6, <json>);
Line 85: Line 116:
INSERT INTO BuildQueue (job, job_type, processor, virtualized)
VALUES (<jobid>, 4,
INSERT INTO BuildQueue (job, job_type)
VALUES (<jobid>, 4)
}}}

This should be enough to get this job onto the build farm. Keep an eye on https://dogfood.launchpad.net/builders where it should appear shortly (and then, after processing, disappear again).

Once the job is successfully completed, any generated templates should appear on the productseries' translations import queue on `dogfood`.

Trying out template generation on dogfood/staging

Right now we have no good test platform for this: staging has no build farm, and dogfood has no codehosting. Expect this page to change a lot as we work and learn. We already have pages about doing this on a development system that may also help.

Internal Soyuz documentation on dealing with dogfood.

Approach

For now, we'll have to live with a split test setup:

  • Push changes to generate jobs on staging.

  • Copy the jobs over to the dogfood database.

  • Let the dogfood build farm generate the templates.

Known Problems

Slave firewalls

The dogfood slaves have to be allowed outgoing http to the staging and production codehosting servers.

Slave failure handling

Soyuz/buildmaster doesn't deal well with the slave failures. When I first got this, the jobs kept being restarted ad infinitum, leaving the builder effectively unusable. This turned out to be because the version on dogfood had been altered for testing. It will go away.

intltool recognition

It looks like gettext has changed, and our check for intltool branches now also accepts some plain gettext branches. This happens with lp:gawk, which pottery mistakenly recognizes as intltool but doesn't process successfully because there is no GETTEXT_PACKAGE definition.

Checklist

Firewall

The slaves in the dogfood build farm must be allowed to make http connections to whatever codehosting server you will be using.

TODO: Ensure this—bug 499407.

Config

Make sure that generating templates is enabled in the Launchpad lazr config.

  • Enabled for dogfood.

  • Enabled on development systems.
  • Being enabled for staging. TODO: what production-configs revision? How will we know it's landed?

The configuration item in question is generate_templates in the [rosetta] section.

Suitable branches

Make sure you have a branch that is pottery-compatible. (We call the code that generates templates from code "pottery"). For testing, it should normally not contain any .pot files. Otherwise we'd just end up importing those instead of ones we generated.

We don't yet know for sure what real-world branches will work right out of the box. We must fix that!

According to wgrant, on many branches, editing configure.ac to set GETTEXT_PACKAGE does the trick. See bug 568355.

You can easily check if a branch will work by generating a template locally. Run the following command from the LP tree in the packages root directory:

scripts/rosetta/pottery-generate-intltool.py

It outputs the template files it creates. It uses the same code as the build slave.

TODO: Find a good real-world sample branch for testing—preferably small but with multiple templates.

Candidate branches:

  • gimp, but it's large.

  • lp:ubuntu/gawk does not use intltool, but pottery seems to think that it does. See bug 568372.

  • lp:libgnomeui: OK, one template

  • lp:gconf: OK, one template

Once you have a suitable branch, you can copy it to staging's codehosting using "bzr push -d lp:<branch> lp://staging/<branch> --use-existing-dir"

Procedure

Set up a product series on both staging and dogfood.

Set up a pottery-compatible branch on staging; set it as the product series' development branch.

Register a branch on dogfood, but do not push it (as dogfood has no codehosting). Set it as the product series' development branch.

Configure translation template imports for both product series.

Push a change to your branch on staging. Wait for it to be scanned.

Once the job is scanned, a triplet of matching records should appear in the staging database: a BranchJob, a Job, and a BuildQueue. They are matched on BranchJob.job, Job.id, and BuildQueue.job. In SQL it should be at the top of...

SELECT *
FROM BranchJob
JOIN Job ON Job.id = BranchJob.job
JOIN BuildQueue ON BuildQueue.job = Job.id
WHERE
    BranchJob.job_type = 6 AND
    BuildQueue.job_type = 4 AND
    Job.status = 0
ORDER BY Job.id DESC;

Copy what you find directly into the dogfood database. Start with the Job:

INSERT INTO Job (status) values (0) RETURNING id;

Note the id that this returns. Now, create a BranchJob using that job id, the id of your branch on dogfood, and a copy of the json_data as created on staging:

INSERT INTO BranchJob (job, branch, job_type, json_data)
VALUES (<jobid>, <branchid>, 6, <json>);

Finally, the BuildQueue entry:

INSERT INTO BuildQueue (job, job_type)
VALUES (<jobid>, 4)

This should be enough to get this job onto the build farm. Keep an eye on https://dogfood.launchpad.net/builders where it should appear shortly (and then, after processing, disappear again).

Once the job is successfully completed, any generated templates should appear on the productseries' translations import queue on dogfood.

Translations/GenerateTemplatesOnTestServers (last edited 2020-09-17 12:48:23 by cjwatson)