Diff for "Translations/Specs/ReportingAPI"

Not logged in - Log In / Register

Differences between revisions 3 and 23 (spanning 20 versions)
Revision 3 as of 2010-02-22 15:24:48
Size: 3463
Editor: dpm
Comment:
Revision 23 as of 2011-02-22 15:34:23
Size: 8727
Editor: dpm
Comment:
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
= Launchpad Translations API = = Launchpad Translations Reporting API =
Line 5: Line 5:
Development of a Launchpad Translations API. This Launchpad enhancement proposal discusses extending the functionality of Launchpad Translations by developing an API which will expose the necessary interfaces to allow read access to translations-related data from Launchpad through [[https://help.launchpad.net/API/launchpadlib|launchpadlib]] for reporting purposes. API access will be provided for both translatable distributions and projects in Launchpad.
Line 7: Line 7:
This Launchpad enhancement proposal discusses extending the functionality of Launchpad Translations by allowing read and write access to translations-related data from Launchpad through launchpadlib. '''On Launchpad:''' https://blueprints.launchpad.net/ubuntu/+spec/community-m-launchpad-translations-reporting-api
Line 9: Line 9:
'''On Launchpad:''' https://blueprints.launchpad.net/rosetta/+spec/api == User stories ==
Line 11: Line 11:
'''As the ''' $PERSON<<BR>>
'''I want ''' $FEATURE<<BR>>
'''so that ''' $BENEFIT<<BR>>
'''As an ''' Ubuntu Translations Coordinator<<BR>>
'''I want ''' to be able to report on the status of Ubuntu translations at the distro level (only default installation source packages)<<BR>>
'''so that ''' I can present the data to the community in a way they can see the real status of translation, track their work, set translation goals, and in short, improve the translation coverage in all languages<<BR>>
Line 15: Line 15:
''Consider clarifying the feature by describing what it is not?'' '''As an ''' OEM Engineer <<BR>>
'''I want ''' to be able to create an automated report on the translation coverage of a set of source packages in a given set of languages<<BR>>
'''so that ''' I can inform a customer and assess whether additional translation work is required before shipping

'''As an ''' Ubuntu translator<<BR>>
'''I want ''' to be able to obtain translations data in an automated way<<BR>>
'''so that ''' I can present and use the data in alternative ways that will help my team better organize translation work
Line 19: Line 25:
''Why are we doing this now?'' Due to other higher priority development work, the Translations component has been lacking a full API for quite some time now.
Line 21: Line 27:
''What value does this give our users? Which users?'' The need of an API arises now from the necessity of [[https://blueprints.launchpad.net/ubuntu/+spec/community-lucid-improving-translation-status-reporting|better reporting the status of Ubuntu translations]]. The main benefit accessing data through launchpadlib can provide to Ubuntu is on the statistics reporting side: currently the main access point to translating Ubuntu is at http://translations.launchpad.net/ubuntu. This view offers statistics to ''all'' translatable packages in the `main` and `restricted` repositories, regardless of distribution. While complete, it does not provide information on how well translated Ubuntu is as a distribution. This information can be very useful to Ubuntu translators in assessing what needs to be translated and on which packages they should concentrate their work. This can also be used to track progress, set up goals, and in short, as an aid to building community around translations. In the same way, this can be useful for commercial customers to assess translation coverage when evaluating to ship Ubuntu in a given language.

As the current focus of Launchpad development is on upstream integration, and as there is no planned UI work in that area, an API appears to be the most adequate way to allow accessing the Launchpad Translations data and present it in alternative ways to suit the stakeholders' needs.
Line 26: Line 34:
 * Ubuntu Desktop Team: [[https://launchpad.net/~arnegoetje|Arne Goetje]]
Line 28: Line 35:

(David: as the drafter of this proposal, I'm regularly in touch with all of the stakeholders)
Line 31: Line 40:
The new behaviour must provide a way to access translations data from Launchpad, either in read (or depending on the value) or write mode through launchpadlib, without the need for an admin to query the database. The new behaviour must provide a way to access translations data from Launchpad, either in read (or depending on the value) or write mode through launchpadlib, without the need for an admin to query the database directly.
Line 33: Line 42:
It must not provide a way to circumvent any security privileges built in the layers above (e.g. Launchpad UI) It must not provide any additional functionality or a way to circumvent any security privileges built in the layers above (e.g. Launchpad UI)
Line 37: Line 46:
The API can be initially subdivided in several areas, which will be exposed separately below. This section will be used to list every separate aspect of reporting, with '''ATTRIBUTES''' lists as the proposals for the direct data to be exposed and '''METHODS''' as indirect data that can be obtained by calling a web app method or another referenced resource
Line 39: Line 48:
=== Reporting === === Language ===
Line 41: Line 50:
==== Languages ==== '''ATTRIBUTES:'''
 * [DONE] Language code
 * [DONE] Language name in English
 * [DONE] Language state (visible or not)
 * [DONE-FUZZY] Number of contributors
 * [DONE] ''Plural forms'' [optional, for checking language support]
 * [DONE] ''Text direction'' [optional]
 * ''The rest of the data at https://translations.launchpad.net/+languages/$LANG'' [optional]
Line 43: Line 59:
Retrieve the list of all languages available in Launchpad. === Language Set ===
Line 45: Line 61:
'''DATA:'''
  * Language code
  * Language name in English
  * Language state (visible or not)
  * Plural forms (optional, for checking language support)
  * Text direction (optional)
'''METHODS:'''
 * [DONE] Get all active languages in Launchpad
 * [DONE] Get all languages known in Launchpad
Line 52: Line 65:
It would be nice to some other data presented on language page https://translations.launchpad.net/+languages/ro , but this is a good start === PO Template ===
Line 54: Line 67:
It would be nice to be able to query just a specific language. For each template the following data would be required:
Line 56: Line 69:
==== PO Templates ====
Retrieve the list of all templates available for a series in Launchpad.
'''ATTRIBUTES''':
 * Template name in Launchpad
 * Priority
 * Length
 * Date last updated
 * Domain name
 * Iscurrent (active or not)
 * Source package associated
 * Source package component (option - to avoid a new query on source package)
 * Is it used in language packs
 * ''In which languages there is a translation available'' [optional]
Line 59: Line 81:
'''DATA''':
 * template name in Launchpad
 * priority
 * length
 * date last updated
 * domain name
 * iscurrent (active or not)
 * source package
 * is in lang pack
 * the list of POFiles associated with this POTemplate
 * in how many languages it is translates (nice to have)
'''METHODS''':
 * The list of POFiles associated with this POTemplate
 * Translation statistics per language for each POFile associated with this POTemplate
Line 71: Line 85:
It would be nice to be able to query a template from a series, based on the Launchpad template name. === PO Files ===
Line 73: Line 87:
==== PO Files ====
Retrieve the list of all PO files for a template in a series.
'''ATTRIBUTES'''
 * Language code
 * Length
Line 76: Line 91:
The list should contain:
 * language code
 * when the translation was last updated
 * number of translated messages
 * number of unreviewed messages
 * number of changed messages
 * who was the last translator
'''METHODS''':
 * A method to display translation statistics:
   * Number of translated messages
   * Number of unreviewed messages
   * Number of changed messages
   * When the translation was last updated
   * Who the last translator was
Line 84: Line 99:
=== Source Package ===
Line 85: Line 101:
=== Import queue === We already have a few exported attributes for source package, but to help in identifying translations moved to univers we also need to know the component for this sourcepackage (https://api.launchpad.net/+apidoc/devel.html#source_package)
Line 87: Line 103:
=== Translation === '''ATTRIBUTES:'''
 * [DONE] Component name as latest_published_component_name (ie: main or universe)

'''METHODS:'''
 * List of all associated PO Templates
 * Translation statistics for each language - per template and aggregated for all templates

=== Series (distribution and product) ===

'''METHODS:'''
 * List of all associated PO Templates
 * Translation statistics for each language - per template and aggregated for all templates
Line 91: Line 118:
''What are the workflows for this feature?''
''Provide mockups for each workflow.''
 * [[../../ApiDevelopment|Previous discussion on the API development]]
 * [[https://bugs.launchpad.net/rosetta/+bug/525371/comments/1|Danilo's thoughts on structuring the work]] for the reporting part:
  1. '''Exposing API for single ''PO file'' statistics''' - a shared interface IRosettaStats will need to be exported through API, so all the other objects which are based on it will get the same API as well.
  1. '''Devising a good API for a single ''PO template'' '''
   * Other than the simple properties, the most interesting bit here is how to get statistics for a bunch PO files attached to a template. If you don't do anything but just export a single method to list out all the PO files, you'll be doing one API call for each of the languages for a PO template and that's going to be very slow. Eg. to get full status export for Ubuntu out of LP you'd need to do some ~400k API calls.
   * A much better solution to that is to provide a method which gives you all related PO file statistics: e.g. `getPOFilesWithStatistics()` which returns tuples of form (`POFile`, `translated_count`, `changed_count`, `unreviewed_count`, `untranslated_count`) directly. FWIW, I'd keep this a separate step as well.
  1. '''IPOTemplateS(ubs)et API''' - as the final step, I'd do an export of `IPOTemplateS(ubs)et` APIs for all the different objects that have them. As another part of this step, you'd probably want to do the similar to above and provide a getPOFilesWithStatistics(language) method on them.
Line 94: Line 126:
'''''You do not have to get the mockups and workflows right at this point. In fact, it is better to have several alternatives, delaying deciding on the final set of workflows until the last responsible moment.''''' The idea behind this workflow is simply to use a gradual approach, which will be easier to get right and get landed, without ending up with a too large a branch to review or follow.
Line 98: Line 130:
''How will we know when we are done?'' Although the intention is to propose a full API, the two main success criteria are the implementation of API functions for reporting and the imports queue, so we will know that we are done when we can report on the status of Ubuntu translations at a distro level (i.e. on a given set of packages, for all Launchpad languages) by having sourced the data trough launchpadlib.
Line 100: Line 132:
''How will we measure how well we have done?'' The success can be measured in how seamlessly this data can be obtained and in whether all requirements in terms of exposed data have been implemented.
Line 104: Line 136:
[[../ApiDevelopment|Previous discussion on the API development]]
[[/ProposedCalls|Proposed API calls]]
 * [[/ProposedCalls|Proposed API calls]]

Launchpad Translations Reporting API

This Launchpad enhancement proposal discusses extending the functionality of Launchpad Translations by developing an API which will expose the necessary interfaces to allow read access to translations-related data from Launchpad through launchpadlib for reporting purposes. API access will be provided for both translatable distributions and projects in Launchpad.

On Launchpad: https://blueprints.launchpad.net/ubuntu/+spec/community-m-launchpad-translations-reporting-api

User stories

As an Ubuntu Translations Coordinator
I want to be able to report on the status of Ubuntu translations at the distro level (only default installation source packages)
so that I can present the data to the community in a way they can see the real status of translation, track their work, set translation goals, and in short, improve the translation coverage in all languages

As an OEM Engineer
I want to be able to create an automated report on the translation coverage of a set of source packages in a given set of languages
so that I can inform a customer and assess whether additional translation work is required before shipping

As an Ubuntu translator
I want to be able to obtain translations data in an automated way
so that I can present and use the data in alternative ways that will help my team better organize translation work

Rationale

Due to other higher priority development work, the Translations component has been lacking a full API for quite some time now.

The need of an API arises now from the necessity of better reporting the status of Ubuntu translations. The main benefit accessing data through launchpadlib can provide to Ubuntu is on the statistics reporting side: currently the main access point to translating Ubuntu is at http://translations.launchpad.net/ubuntu. This view offers statistics to all translatable packages in the main and restricted repositories, regardless of distribution. While complete, it does not provide information on how well translated Ubuntu is as a distribution. This information can be very useful to Ubuntu translators in assessing what needs to be translated and on which packages they should concentrate their work. This can also be used to track progress, set up goals, and in short, as an aid to building community around translations. In the same way, this can be useful for commercial customers to assess translation coverage when evaluating to ship Ubuntu in a given language.

As the current focus of Launchpad development is on upstream integration, and as there is no planned UI work in that area, an API appears to be the most adequate way to allow accessing the Launchpad Translations data and present it in alternative ways to suit the stakeholders' needs.

Stakeholders

(David: as the drafter of this proposal, I'm regularly in touch with all of the stakeholders)

Constraints

The new behaviour must provide a way to access translations data from Launchpad, either in read (or depending on the value) or write mode through launchpadlib, without the need for an admin to query the database directly.

It must not provide any additional functionality or a way to circumvent any security privileges built in the layers above (e.g. Launchpad UI)

Subfeatures

This section will be used to list every separate aspect of reporting, with ATTRIBUTES lists as the proposals for the direct data to be exposed and METHODS as indirect data that can be obtained by calling a web app method or another referenced resource

Language

ATTRIBUTES:

  • [DONE] Language code
  • [DONE] Language name in English
  • [DONE] Language state (visible or not)
  • [DONE-FUZZY] Number of contributors
  • [DONE] Plural forms [optional, for checking language support]

  • [DONE] Text direction [optional]

  • The rest of the data at https://translations.launchpad.net/+languages/$LANG [optional]

Language Set

METHODS:

  • [DONE] Get all active languages in Launchpad
  • [DONE] Get all languages known in Launchpad

PO Template

For each template the following data would be required:

ATTRIBUTES:

  • Template name in Launchpad
  • Priority
  • Length
  • Date last updated
  • Domain name
  • Iscurrent (active or not)
  • Source package associated
  • Source package component (option - to avoid a new query on source package)
  • Is it used in language packs
  • In which languages there is a translation available [optional]

METHODS:

  • The list of POFiles associated with this POTemplate
  • Translation statistics per language for each POFile associated with this POTemplate

PO Files

ATTRIBUTES

  • Language code
  • Length

METHODS:

  • A method to display translation statistics:
    • Number of translated messages
    • Number of unreviewed messages
    • Number of changed messages
    • When the translation was last updated
    • Who the last translator was

Source Package

We already have a few exported attributes for source package, but to help in identifying translations moved to univers we also need to know the component for this sourcepackage (https://api.launchpad.net/+apidoc/devel.html#source_package)

ATTRIBUTES:

  • [DONE] Component name as latest_published_component_name (ie: main or universe)

METHODS:

  • List of all associated PO Templates
  • Translation statistics for each language - per template and aggregated for all templates

Series (distribution and product)

METHODS:

  • List of all associated PO Templates
  • Translation statistics for each language - per template and aggregated for all templates

Workflows

  • Previous discussion on the API development

  • Danilo's thoughts on structuring the work for the reporting part:

    1. Exposing API for single PO file statistics - a shared interface IRosettaStats will need to be exported through API, so all the other objects which are based on it will get the same API as well.

    2. Devising a good API for a single PO template

      • Other than the simple properties, the most interesting bit here is how to get statistics for a bunch PO files attached to a template. If you don't do anything but just export a single method to list out all the PO files, you'll be doing one API call for each of the languages for a PO template and that's going to be very slow. Eg. to get full status export for Ubuntu out of LP you'd need to do some ~400k API calls.
      • A much better solution to that is to provide a method which gives you all related PO file statistics: e.g. getPOFilesWithStatistics() which returns tuples of form (POFile, translated_count, changed_count, unreviewed_count, untranslated_count) directly. FWIW, I'd keep this a separate step as well.

    3. IPOTemplateS(ubs)et API - as the final step, I'd do an export of IPOTemplateS(ubs)et APIs for all the different objects that have them. As another part of this step, you'd probably want to do the similar to above and provide a getPOFilesWithStatistics(language) method on them.

The idea behind this workflow is simply to use a gradual approach, which will be easier to get right and get landed, without ending up with a too large a branch to review or follow.

Success

Although the intention is to propose a full API, the two main success criteria are the implementation of API functions for reporting and the imports queue, so we will know that we are done when we can report on the status of Ubuntu translations at a distro level (i.e. on a given set of packages, for all Launchpad languages) by having sourced the data trough launchpadlib.

The success can be measured in how seamlessly this data can be obtained and in whether all requirements in terms of exposed data have been implemented.

Thoughts?

Translations/Specs/ReportingAPI (last edited 2011-02-22 15:34:23 by dpm)