Translations/Specs/MessageSharing/Migration

Not logged in - Log In / Register

Revision 13 as of 2009-03-17 10:56:01

Clear message

Message Sharing Migration

Part of the message sharing project.

Assumptions

Code changes

To be landed before migration:

Note: When clearing is_current in validate_is_current, converge the TM that's having the flag cleared.

Migration steps

Migration consists of these phases:

Initial Cleanup

Before migrating, we should delete all non-current POTemplates. This eliminates some complications in migration.

Before we can delete a POTemplate, we must delete any rows that refer to it:

  1. Clear TranslationMessage.pofile if the column still exists. We won't be using it anymore.

  2. Delete all TranslationMessages that refer to the template in their potemplate fields.

  3. Find all TranslationTemplateItems that refer to the template, and delete them.

  4. Find any POTMsgSets that the deleted TranslationTemplateItems referred to, and that have no other TranslationTemplateItems referring to them.

  5. Delete any TranslationMessages attached to those POTMsgSets.

  6. Delete those POTMsgSets.

  7. Delete all POFiles that refer to the template. Any references in POFileTranslator should have gone away when we deleted the TranslationMessages.

  8. Delete the POTemplate.

Merge POTMsgSets

At the end of this phase, TranslationMessages will still be mostly diverged, but they'll be sharing POTMsgSets.

This phase requires at least a freeze on imports.

It's best to go through this separately per Ubuntu release, and maybe once for everything else. We must stop imports for whatever product or distroseries we're merging, which we can do individually for each Ubuntu series.

Commit transactions after every equivalence class of POTemplates. Keep track of which ones are done, so we can restart etc.

Steps:

  1. For each equivalence class of POTMsgSets, merge all TranslationTemplateItems into those of the representative POTMsgSet.

  2. For each converged current or imported TranslationMessage:

    • - if the representative POTMsgSet already has another converged current/imported TranslationMessage for the same language/variant, clear the current/imported flag.

  3. For each TranslationMessage:

    • - replace potmsgset with its representative POTMsgSet.

  4. From each POTMsgSet equivalence class:

    • - delete all members but the representative one.

Merge TranslationMessages

This phase operates only on TranslationMessage, and can be done at leisure.

Use TranslationMessage.converge() as defined above. Run it on all TranslationMessages in distroseries/productseries that have translation focus, then on all other series..

Database constraints

Once the assumptions are fulfilled:

  • If we still have POFile, drop all constraints involving TranslationMessage.pofile.

After rolling out code changes:

After migration:

  • On TranslationMessage:

    • potemplate is null where not (is_current or is_imported)
    • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, -1), COALESCE(msgstr0, -1), …)
    • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, -1)) where is_current is true
    • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, -1)) where is_imported is true

Checks

Some things we should check regularly after migration:

  • Due to race conditions it may occasionally be possible for a diverged message to have the same potmsgset and translations as a shared one. We can only guard against that efficiently in python, and run routine checks/cleanups on the database.
    •     SELECT
              diverged_tm.id,
              converged_tm.id,
              diverged_tm.is_current,
              diverged_tm.is_imported
          FROM TranslationMessage diverged_tm
          JOIN TranslationMessage converged_tm ON
              diverged_tm.potmsgset = converged_tm.potmsgset AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1);
    • Delete diverged_tm first to avoid running into database constraints.
    • "OR" diverged_tm's is_current/is_imported into those of converged_tm.
  • Every (potmsgset, potemplate) combination in TranslationMessage must occur in TranslationTemplateItem.