Message Sharing Migration

Part of the message sharing project.

Assumptions

• message-sharing-populate has run.
• Codebase is fully message-sharing-enabled.
• Schema elements that are to be removed are either gone or no longer used.

• NO LONGER VALID: Imported messages are always shared. Danilo's current branch no longer requires this.

Migration steps

Migration consists of these phases:

• Initial Cleanup (optional)
• Code changes

• Merge POTMsgSets

• Merge TranslationMessages

• Restore POFileTranslator

Code changes

To be landed before migration:

• (Pseudocode, any resemblance to Python is pure coincidence) This still assumes that imported messages are shared, which is no longer necessary.

  class TranslationMessage:
      # ...
      def converge(self):
          """Make this message shared if possible, or merge it into an existing shared one."""
          if self.potemplate is None: return
  
          shared = TranslationMessage.get(potemplate=None, self.potmgset, self.language, self.variant, self.translations)
          current = TranslationMessage.get(potemplate=None, self.potmsgset, self.language, self.variant, is_current=True)
          imported = TranslationMessage.get(potemplate=None, self.potmsgset, self.language, self.variant, is_imported=True)
  
          if shared:
              # There's a shared message matching this one.  Try to merge.
              if self.is_imported and imported is None:
                  # Bequeath is_imported flag to shared equivalent.
                  shared.is_imported = True
              if self.is_current and current is None:
                  # Bequeath is_current flag to shared equivalent.
                  self.is_current = False
                  shared.is_current = True
              # This message is only worth keeping if it's a diverged current message.
              if not (self.is_current and not shared.is_current): self.destroy()
          else:
              # No matching shared message yet, so become one.
  
              # If we're about to converge: don't supplant an existing shared imported
              # message.  If we're staying diverged: don't permit imported messages to
              # be diverged.
              if imported is not None: self.is_imported = False
  
              # Converge, unless this is to be a diverged current message.
              if current is None or not self.is_current: self.potemplate = None

Note: When clearing is_current in validate_is_current, converge the TM that's having the flag cleared.

Initial Cleanup

Before migrating, we should probably delete all non-current POTemplates. This saves us from having to worry about them.

Before we can delete a POTemplate, we must delete any rows that refer to it:

1. Clear TranslationMessage.pofile if the column still exists. We won't be using it anymore.

2. Delete all TranslationMessages that refer to the template in their potemplate fields.

3. Find all TranslationTemplateItems that refer to the template, and delete them.

4. Find any POTMsgSets that the deleted TranslationTemplateItems referred to, and that have no other TranslationTemplateItems referring to them.

5. Delete any TranslationMessages attached to those POTMsgSets.

6. Delete those POTMsgSets.

7. Delete all POFiles that refer to the template. Any references in POFileTranslator should have gone away when we deleted the TranslationMessages.

8. Delete the POTemplate.

Merge POTMsgSets

At the end of this phase, TranslationMessages will still be mostly diverged, but they'll be sharing POTMsgSets.

This phase requires at least a freeze on imports.

It's best to go through this separately per Ubuntu release, and maybe once for everything else. We must stop imports for whatever product or distroseries we're merging, which we can do individually for each Ubuntu series.

Commit transactions after every equivalence class of POTemplates. Keep track of which ones are done, so we can restart etc.

• Define equivalence class of POTemplate as belonging to POTemplates of the same name (not translation domain?) within either the same Product or the same DistroSeries.

• Define equivalence class of POTMsgSets as being attached (by TranslationTemplateItem) to POTemplates of the same equivalence class, and having the same (msgid_singular, COALESCE(msgid_plural, -1), COALESCE(context, )).

• Define the representative POTMsgSet in an equivalence class as the one in the series that has translation focus, if any; or failing that, the one with the highest id.

Pseudocode:

def get_key(potmsgset):
    return (potmsgset.msgid_singular, potmsgset.msgid_plural, potmsgset.context)


def merge_potmsgsets(potemplates):
    # Sort potemplates from "most representative" to "least representative."
    potemplates.sort(cmp=template_precedence)

    representatives = {}
    subordinates = {}

    # Figure out representative potmsgsets and their subordinates.
    for template in potemplates:
        for potmsgset in template.potmsgsets:
            key = get_key(potmsgset)
            if key not in representatives: representatives[key] = potmsgset
            representative = representatives[key]
            if representative in subordinates: 
                subordinates[representative].append(potmsgset)
            else:
                subordinates[representative] = []

    for representative, potmsgsets in subordinates.iteritems():
        for subordinate in potmsgsets:
            merge_translationtemplateitems(subordinate, representative)

            for message in subordinate.translation_messages:
                if message.potemplate is None:
                    # Guard against multiple imported messages.
                    if message.is_imported:
                        imported = representative.getImportedMessage(message.language, message.variant)
                        if imported is not None: message.is_imported = False
                    # Guard against multiple shared current messages.
                    if message.is_current:
                        current = representative.getCurrentMessage(message.language, message.variant, potemplate=None)
                        if current is not None: message.is_current = False
                message.potmsgset = representative
            subordinate.destroy()

• On TranslationMessage:

  • potemplate is null where is_imported
  • potemplate is null where not is_current

  • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, ), COALESCE(msgstr0, -1), …)

  • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, )) where is_current is true

  • unique (potmsgset, COALESCE(potemplate, -1), language, COALESCE(variant, )) where is_imported is true

Checks

Some things we should check regularly after migration:

• Due to race conditions it may occasionally be possible for a diverged message to have the same potmsgset and translations as a shared one. We can only guard against that efficiently in python, and run routine checks/cleanups on the database.

  • Any rows we get out of this would be redundant:

        SELECT
            diverged_tm.id,
            converged_tm.id,
            diverged_tm.is_current,
            diverged_tm.is_imported
        FROM TranslationMessage diverged_tm
        JOIN TranslationMessage converged_tm ON
            diverged_tm.potmsgset = converged_tm.potmsgset AND
            diverged_tm.language = converged_tm.language AND
            diverged_tm.potemplate IS NOT NULL AND
            COALESCE(diverged_tm.variant, '') = COALESCE(converged_tm.variant, '') AND
            COALESCE(diverged_tm.msgstr0, -1) = COALESCE(converged_tm.msgstr0, -1) AND
            COALESCE(diverged_tm.msgstr1, -1) = COALESCE(converged_tm.msgstr1, -1) AND
            COALESCE(diverged_tm.msgstr2, -1) = COALESCE(converged_tm.msgstr2, -1) AND
            COALESCE(diverged_tm.msgstr3, -1) = COALESCE(converged_tm.msgstr3, -1) AND
            COALESCE(diverged_tm.msgstr4, -1) = COALESCE(converged_tm.msgstr4, -1) AND
            COALESCE(diverged_tm.msgstr5, -1) = COALESCE(converged_tm.msgstr5, -1)
        WHERE
            converged_tm.potemplate IS NULL AND
            converged_tm.is_current;

  • Delete diverged_tm first to avoid running into database constraints.
  • "OR" diverged_tm's is_current/is_imported into those of converged_tm.

• Every (potmsgset, potemplate) combination in TranslationMessage must occur in TranslationTemplateItem.