Translations/Specs/MessageSharing/Migration

Not logged in - Log In / Register

Revision 11 as of 2009-03-17 10:34:10

Clear message

Message Sharing Migration

Part of the message sharing project.

Assumptions

Code changes

To be landed before migration:

class TranslationMessage:
    # ...
    def converge(self):
        """Make this message shared if possible, or merge it into an existing shared one."""
        shared = TranslationMessage.get(potemplate=None, self.potmgset, self.language, self.variant, self.translations)
        current = TranslationMessage.get(potemplate=None, self.potmsgset, self.language, self.variant, is_current=True)
        imported = TranslationMessage.get(potemplate=None, self.potmsgset, self.language, self.variant, is_imported=True)

        if shared:
            # There's a shared message matching this one.  Try to merge.
            if self.is_imported and imported is None:
                # Bequeath is_imported flag to shared equivalent.
                shared.is_imported = True
            if self.is_current and current is None:
                # Bequeath is_current flag to shared equivalent.
                self.is_current = False
                shared.is_current = True
            # This message is only worth keeping if it's a diverged current message.
            if not (self.is_current and not shared.is_current): self.destroy()
        else:
            # No matching shared message yet, so become one.

            # If we're about to converge: don't supplant an existing shared imported
            # message.  If we're staying diverged: don't permit imported messages to
            # be diverged.
            if imported is not None: self.is_imported = False

            # Converge, unless this is to be a diverged current message.
            if current is None or not self.is_current: self.potemplate = None

Note: When clearing is_current in validate_is_current, converge the TM that's having the flag cleared.

Migration steps

Migration consists of these phases:

Initial Cleanup

Before migrating, we should delete all non-current POTemplates. This eliminates some complications in migration.

Before we can delete a POTemplate, we must delete any rows that refer to it:

  1. Clear TranslationMessage.pofile if the column still exists. We won't be using it anymore.

  2. Delete all TranslationMessages that refer to the template in their potemplate fields.

  3. Find all TranslationTemplateItems that refer to the template, and delete them.

  4. Find any POTMsgSets that the deleted TranslationTemplateItems referred to, and that have no other TranslationTemplateItems referring to them.

  5. Delete any TranslationMessages attached to those POTMsgSets.

  6. Delete those POTMsgSets.

  7. Delete all POFiles that refer to the template. Any references in POFileTranslator should have gone away when we deleted the TranslationMessages.

  8. Delete the POTemplate.

Merge POTMsgSets

At the end of this phase, TranslationMessages will still be mostly diverged, but they'll be sharing POTMsgSets.

This phase requires at least a freeze on imports.

It's best to go through this separately per Ubuntu release, and maybe once for everything else. We must stop imports for whatever product or distroseries we're merging, which we can do individually for each Ubuntu series.

Commit transactions after every equivalence class of POTemplates. Keep track of which ones are done, so we can restart etc.

Steps:

  1. For each equivalence class of POTMsgSets, merge all TranslationTemplateItems into those of the representative POTMsgSet.

  2. For each converged current or imported TranslationMessage:

    • - if the representative POTMsgSet already has another converged current/imported TranslationMessage for the same language/variant, clear the current/imported flag.

  3. For each TranslationMessage:

    • - replace potmsgset with its representative POTMsgSet.

  4. From each POTMsgSet equivalence class:

    • - delete all members but the representative one.

Merge TranslationMessages

This phase operates only on TranslationMessage, and can be done at leisure.

  1. Converge all TMs where is_current is false and is_imported is false.
  2. Converge TMs where is_current or is_imported, and where TM.potmsgset.potemplate.distroseries has translation focus.
  3. Converge TMs where is_current or is_imported, and where TM.potmsgset.potemplate.productseries has translation focus.
  4. Clean up POTMsgSets without TranslationTemplateItems attached.

Database constraints

Once the assumptions are fulfilled:

  • If we still have POFile, drop all constraints involving TranslationMessage.pofile.

After rolling out code changes:

After migration:

  • On TranslationMessage:

    • potemplate is null where not (is_current or is_imported)
    • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, -1), COALESCE(msgstr0, -1), …)
    • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, -1)) where is_current is true
    • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, -1)) where is_imported is true

Checks

Some things we should check regularly after migration:

  • Due to race conditions it may occasionally be possible for a diverged message to have the same potmsgset and translations as a shared one. We can only guard against that efficiently in python, and run routine checks/cleanups on the database.
    •     SELECT
              diverged_tm.id,
              converged_tm.id,
              diverged_tm.is_current,
              diverged_tm.is_imported
          FROM TranslationMessage diverged_tm
          JOIN TranslationMessage converged_tm ON
              diverged_tm.potmsgset = converged_tm.potmsgset AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
              COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1);
    • Delete diverged_tm first to avoid running into database constraints.
    • "OR" diverged_tm's is_current/is_imported into those of converged_tm.
  • Every (potmsgset, potemplate) combination in TranslationMessage must occur in TranslationTemplateItem.