Language Packs

[nacin]

Implement language packs for core, plugins, and themes.

Inspiration and code can hopefully be derived from the corresponding GSoC project.

This will require quite a bit of work in GlotPress, on api.wordpress.org, and in core. I will take point, but assistance will be needed. A number of decisions will need to be made. I will begin designing a document for what exactly needs to be done over the next week.

[nacin]

Two files are currently hard-coded and committed into SVN by translators: setup-config.php, and wp-config-sample.php.

18200.diff​ implements WP_I18N in setup-config.php. It does not yet touch the default constant values, which must match wp-config-sample.php (and code exists on WP.org to verify this).

The goal would be for these strings to be implemented directly into GlotPress and for wp-config-sample.php to then be filled out in Rosetta. (Rosetta is the code and associated tools that power the local language sites, such as ​http://pt.wordpress.org/.) This will then remove the SVN requirement for nearly all core packages. (I haven't gotten into $locale.php or special plugins yet, but we will.)

[nacin]

A sanity check on my syntax for this patch would be appreciated. Untested and I may have missed a string.

[SergeyBiryukov]

Replying to nacin:

Two files are currently hard-coded and committed into SVN by translators: setup-config.php, and wp-config-sample.php.

There's also:

• readme.html
• $wp_default_secret_key in default-constants.php (not required, but prevents the admin area from breaking after installing some plugins, see #14024 and the ​post on WP Polyglots)

Replying to nacin:

A sanity check on my syntax for this patch would be appreciated.

Tested installation, works fine. Minor correction in line 85:

'WordPress › Setup Configuration File'/*WP_I18N_SETUP_CONFIG_EXISTS*/;

Should be:

'WordPress › Setup Configuration File'/*/WP_I18N_SETUP_TITLE*/;

[nacin]

Re-opened #18180 and will continue setup-config.php conversation there.

[ocean90]

Related: #16775

[dimadin]

Related: #15977

[jane]

It was decided at last week's dev chat to punt this because we are past freeze and it's not quite done. Hopefully it will be ready as soon as we open the 3.4 branch.

[scribu]

Related: #18960

[SergeyBiryukov]

Related: #19832

[dd32]

refresh of last alterations

[dimadin]

Oh, dd32, why you didn't post a comment, we could have never seen this patch? :)

Anyway, I'll try to summarize what this patch does so you correct me where I'm wrong:

• On core/plugins/themes check for updates, we now check for languages too.
• Checks for languages are done by searching for language files in WP_LANG_DIR, from which is then PO-Revision-Date taken to be sent to api.wp.org.
• When any install/upgrade of core/plugin/theme is done, rechecks for updates are forced for everything so that there are fresh results against latest versions.
• After that, upgrades for available languages updates for everything (core/plugins/themes) are forced.

[dd32]

Oh, dd32, why you didn't post a comment, we could have never seen this patch? :)

Because the patch is more than useless without the WordPress.org infrastructure being setup (And it wasn't set up, and still isn't ready). Those that needed to know there's a patch here knew about it :)

The overview of what you've said sounds about right for the process it'd be doing.

[nacin]

Considering this for 3.5 for bundled themes and importers.

[dd32]

Refreshed patch - completely untested

[ocean90]

Do we have the WordPress.org infrastructure now?

[dd32]

Language Packs are currently blocked by needing WordPress.org development, Shifting out of 3.5 since it won't be included.

[dd32]

In 22346:

Theme Translations: Allow for theme pomo files to be loaded from WP_LANG_DIR/themes/{$domain}-{$locale}.(p|m)o.
This directory format is what we have chosen for Language Packs (See #18200), but which is currently delayed.

By making this change, we can ship localised theme files within core for bundled themes, and avoid the issues associated with Theme Updates overwriting/removing the language files.

Fixes #18960

[dimadin]

Attached patch implements changes from [22346] to plugins.

[dimadin]

Non-blocking

I disagree that this needs to wait WordPress.org infrastructure before inclusion in core. The way this should work based on proposed patch is that when there is no WordPress.org infrastructure it won’t return anything language related so it would simply work as before.

This means that infrastructure might get ready late in release cycle, or even after release, and in that time we’ll have core side done and tested. Core support doesn’t require API (api.wordpress.org) support, it’s only required if you want to have it as an official feature.

Probably, in beginning there will be need for some change in API’s response so that response can be able to hold information for a few language-core/plugin/theme combinations that would have pre-made static language packages for testing purposes. This shouldn’t require too much changes.

Testing

I tested patch by @dd32. Everything in install/upgrade of core/plugins/themes worked as usual so there is nothing breaking. For packs, even if response had language information, nothing would happen since patch doesn’t save that data.

So I modified it (see attached patch) with presumption that structure of API responses is same. Response for core is already structured so that there are a couple of keys with different content and it would simply access one more for languages. Plugins and themes responses offer arrays where first level keys are names of plugins/themes so it just checks for language_updates key and unsets it.

Then I forced response to hold language information with pre-made packages (see attached plugin).

With language_updates available, upgrader would download and extract language files as expected.

Improvements

There are possible improvements to this core workflow:

• API responses for plugins/themes should change to be in line with core one (current array of updates can be under offers key, with languages under language_updates).
• API requests should have one more key along installed_languages that carries language codes so that site can get language files even if there aren’t installed language files.
• On unistallation of plugin/theme, associate language files should be deleted too.
• Action links on upgrader screen should be maybe shown after language updates are done to avoid possible interruption caused by user’s click on those links.

GlotPress

On GP side, I added some related patches, like introducing hooks when translation is changed (to rebuild package) or projects branching. I have other ideas for GP improvements but that’s blocked by lack of any development over there.

Conclusion

I propose that core support for language packs lands early in 3.6 cycle. This would give enough time for testing and shift attention to WordPress.org and GlotPress since they need a lot more changes.

[dimadin]

Refreshed patch, tested as described in comment:ticket:18200:43

[dimadin]

Plugin used to test attachment:ticket:18200:lp.5.diff​ as described in comment:ticket:18200:43

[nacin]

In 23912:

WP_Upgrader: Add upgrader_process_complete hooks and add a abort_if_destination_exists flag (default is true). props dd32. see #18200.

[wonderboymusic]

these are all marked 3.7-early

[dimadin]

I've refreshed patch against latest revision.

Note that attachment:ticket:18200:no_plugin_overwrite.patch​ doesn't need refreshing and I see no reason why this can't go immediately since we have this feature for themes since 3.5.

[nacin]

#25044 was marked as a duplicate.

[nacin]

In 25059:

Allow for plugin translations to be loaded from WP_LANG_DIR/plugins/$domain-$locale.mo.

props dimadin.
see #18200.

[nacin]

In 25520:

Send installed language data to the plugin and theme update-check endpoints.

Introduces wp_get_installed_language_data() and wp_get_pomo_file_data().

see #18200.

[nacin]

In 25536:

Accept and store translations data from the plugin and theme update check endpoints.

Send site locale. Rename wp_get_installed_language_data() to wp_get_installed_translations().

see #18200.

[rmccue]

In 25536, shouldn't $new_update->translations also be set to an empty array if the response fails (similar to $new_update->response)?

[ocean90]

Replying to rmccue:

In 25536, shouldn't $new_update->translations also be set to an empty array if the response fails (similar to $new_update->response)?

A bit related:

Notice: Undefined index: translations in wp-includes/update.php on line 230

[dd32]

In 25544:

Plugin Update API: Set the translations key to an empty array when the API is unavailable. Props rmmcue. See #18200

[dd32]

Notice: Undefined index: translations in wp-includes/update.php on line 230

That should be fixed now I think, for about half an hour after that initial commit the API wasn't returning the field. It is now, so it shouldn't hit that condition.

[nacin]

In 25566:

Add a language pack upgrader class.

At the conclusion of any upgrade, after the transients are refreshed from the API, pending translations are downloaded and installed to wp-content/languages.

props dd32.
see #18200.

[dd32]

In 25568:

Language Packs: Integrate Language Packs into the Auto-upgrader. See #18200 See #22704

[netweb]

​#bbPress2437 Test and ensure support within bbPress plugin for WordPress Core ​#WP18200 :)

[nacin]

What delete_plugins() might need to delete language files. Let's ignore this entirely for now. If we wanted to, we could clear up unused language files on a future update.

[nacin]

In 25634:

Filters for locale in wp_update_plugins() and wp_update_themes(). see #18200.

[dd32]

In 23912: WP_Upgrader: Add upgrader_process_complete hooks and add a abort_if_destination_exists flag (default is true). props dd32. see #18200.

I just noticed that upgrader_process_complete passes the 3rd arguement as $plugin or $theme (ie. the item to update), where as core passes $result, the result of the upgrade process.
As a result, only Core upgrades are able to detect if the upgrade failed.

I noticed this because Upon a core upgrade failing, Language packs proceed to try to update themselves, thus, a method to detect core, plugin, and theme update failures via the filter would be good.

[nacin]

In 25652:

Send back core translation information to the API. Update wp_get_installed_translations() to work with core translations. see #18200.

[nacin]

Replying to dd32:

In 23912: WP_Upgrader: Add upgrader_process_complete hooks and add a abort_if_destination_exists flag (default is true). props dd32. see #18200.

I just noticed that upgrader_process_complete passes the 3rd arguement as $plugin or $theme (ie. the item to update), where as core passes $result, the result of the upgrade process.
As a result, only Core upgrades are able to detect if the upgrade failed.

I noticed this because Upon a core upgrade failing, Language packs proceed to try to update themselves, thus, a method to detect core, plugin, and theme update failures via the filter would be good.

I think we should shift $plugin and $theme to argument 4, and make $result always the third argument. Or, we could set the $package to be the third argument for core updates, and make $result always the fourth argument. Either way.

For bulk plugins/themes, is a single $result all we need? Or is it an array of results? I think we should still try to update languages even when something goes wrong in bulk.

[Chouby]

In wp_get_installed_translations, the regex [A-Za-z_]{2,6} for detecting the locale seems a bit restrictive. In my plugin dev, I encountered an issue with this rule with German for which two languages packs are distributed: de_DE as standard and de_DE_Sie for the formal language. See ​http://wpde.org/download/sprachdateien/ . I am not aware of any other exception.

[nacin]

Replying to Chouby:

In wp_get_installed_translations, the regex [A-Za-z_]{2,6} for detecting the locale seems a bit restrictive. In my plugin dev, I encountered an issue with this rule with German for which two languages packs are distributed: de_DE as standard and de_DE_Sie for the formal language. See ​http://wpde.org/download/sprachdateien/ . I am not aware of any other exception.

As far as I can tell, this is extremely wrong. However, we could potentially drop the 6 there, and just match anything. As long as they are only using underscores as delimiters, we should be okay, as a domain that matches a WP.org plugin or theme slug (and ideally any plugin or theme) will always use hyphens.

[pavelevap]

Some questions:

1) I updated one English plugin and there were also automatic language updates/installs of all default themes? I did not want to update/download languages for themes. And it was development SVN repository and I did not expected language downloads. Please do not run language downloads when user does not want (and expect) it.

2) So, every user can update language from GlotPress at any time? And what about when translation will be work in progress? There will be many different versions of translation? There should be any tool (checkbox) for validators to enable project as downloadable for language packs. Or it will be mess. I will validate one day 50 % strings for Twenty Fourteen and many users will download this translation. There will be many support questions "Why is this in English?" Next day I will validate next 20 % strings and all users should redownload? Or language update will run automatically?

[markoheijnen]

I think the first point wouldn't be much of an issue. The regular user doesn't know what he expects or wants. So updating languages at the point is fine.

Also the second thing wouldn't be much of an issue I guess. There should be a threshold for sure like 80% need to be translated. Also for the themes it will not be much of an issue since they will be translated during development. So if you using the theme during development then you do ask for it.

[pavelevap]

Ad 1) It is a serious issue, I guess. For example I do not want to use localized version of any plugin/theme. Yes, I can delete it now, but it will be back during several hours? It will be annoying for many users... And what about when I use my own localization edited for special purposes?

Ad 2) Automatic threshold is not a good idea, it should depend on validators. Sometimes can be only 30 % translated (for example only frontend) and it should be delivered to users. And sometimes there are 80 % translated strings and localization is not completed for users. Please, do not use fully automatic approach...

[markoheijnen]

1) isn't an issue then since it should not download localized versions when you don't have any. The one that it would overwrite your own version will be an issue. That said you should never willing to do so.

2) Even a validator can't really do something about it. Most people translate strings and validate them without knowing where it is used in the plugin/theme. So the only good way is a treshhold or it only get downloaded when all strings are translated since only then it's completed.

[alex-ye]

Replying to pavelevap:

Some questions:

1) I updated one English plugin and there were also automatic language updates/installs of all default themes? I did not want to update/download languages for themes. And it was development SVN repository and I did not expected language downloads. Please do not run language downloads when user does not want (and expect) it.

I know some people who really need this, anyway I think they are 20% or maybe less. so we can make a plugin to manage the Langauge Packs downloads.

2) So, every user can update language from GlotPress at any time? And what about when translation will be work in progress? There will be many different versions of translation? There should be any tool (checkbox) for validators to enable project as downloadable for language packs. Or it will be mess. I will validate one day 50 % strings for Twenty Fourteen and many users will download this translation. There will be many support questions "Why is this in English?" Next day I will validate next 20 % strings and all users should redownload? Or language update will run automatically?

1+

[pavelevap]

Ad 1) No, it is a serious issue. Examples:

a) I have my own language file cs_CZ.mo in theme directory (or using filter for textdomain). But Updater installs its own theme-cs_CZ.mo file in wp-content/languages/themes directory. This file is not used then, but I do not want this file on my server.

b) There is a plugin which somebody translated and validated with the help of automatic translator and it is not possible to use it on production website. I want to use this plugin in original language, but it is not possible by default. I have to use translated version or use some (not existing now) plugin.

Yes, plugin for managing translations would be helpfull, but I am not sure why should I use plugin for things which are broken by WordPress?

Ad 2) No, treshold is really not a good idea. I am maintaining Czech localization for many years. Everytime I am looking for context because much more better is original English than wrong and messy translations.

Example: I will work on Twenty Fourteen localization and there will be 80 % ready. Then I will be offline for two days. Then I will decide to make some important changes. But there will be many people with older (and wrong) language versions. Validators should have at least checkbox for enabling particular project for downloads. Validators are usually experienced people and there is no need to use automatic treshold to replace their opinion. Or should I translate offline with Poedit and only import final translation? It is against collaborative idea of GlotPress, but I do not want to use not completed translation...

Or Upgrader will automatically check downloaded language files everytime and update them if there will be any change in GlotPress?

[markoheijnen]

You mixing point 1 into something different or it wasn't fully explained in the beginning.

Changing plugin files means when you updating them they will be overwritten. That is the same case with language files. If you don't want that then you need to build a plugin thats prevents that. I'm unsure if a md5 hash can help with that.

Not sure what you mean with 1b.

The checkbox would most likely not going to work too. It will be on by default and a few people really going to care about it. And I'm unsure if validators are experienced people. I'm sure you are and I know from a few other languages that they are really strict too but I also know where things get really fast done. What means no checking on context and where it will be used.

[pavelevap]

Yes, maybe, I am still testing and thinking, sorry...

Ad 1a) Yes, but many people know about it now. They use theme, make own edits, own translations and do not care about future updates (actually they will never happen). Instead of md5 hash we could check if there is any other translation loaded (outside wp-content/languages) before downloading language files?

Ad 1b) I do not want to use validated translation with low quality, but original English version.

Ad 2) Checkbox would work for people who want to use it. When I know that translation will be ready in several days, I can turn it off until it is done. It is not a big deal, I guess? Or it can be off by default to make people care about it. Now we can wait for completion and release translation with "big blue button" when needed. It should stay as it is, I hope people are still better then machines :-)

[alex-ye]

Replying to markoheijnen:

The checkbox would most likely not going to work too. It will be on by default and a few people really going to care about it. And I'm unsure if validators are experienced people. I'm sure you are and I know from a few other languages that they are really strict too but I also know where things get really fast done. What means no checking on context and where it will be used.

I suggest to add a select option for validators to enable project as

• "Ready When 100%" (default)

Means that the Language Packs should download the translations if they are 100% only.

• "Ready"

I think it will helpful.. because not all the translations are 100% and it should be an ability to ignore that, since anyone can download it directly from the GlotPress!

• "Not Ready"

In the other side it should be an ability to avoid the auto-downloads even if the translations are 100%... I need this! for example when we finish the first-draft translations we spent more time to double-check the Grammar,Syntax,"Tashkeel"...etc

[pavelevap]

alex-ye: +1

[ocean90]

When updating a plugin and translation files are updated too, the action links can be hard to find, see ​http://cl.ly/RlBq.

[Chouby]

When making an update, languages files are *all* automatically updated at the latest revision (clearly visible in the picture of @ocean90). Which means, for example, that the user might for some reason keep an old theme or plugin (let say Twenty Ten 1.5), but Twenty Ten languages files are updated to 1.6. If some string has changed or has been removed in the latest revision, it won't be translated anymore.

[nacin]

Replying to Chouby:

When making an update, languages files are *all* automatically updated at the latest revision (clearly visible in the picture of @ocean90). Which means, for example, that the user might for some reason keep an old theme or plugin (let say Twenty Ten 1.5), but Twenty Ten languages files are updated to 1.6. If some string has changed or has been removed in the latest revision, it won't be translated anymore.

Thankfully, it's a bit smarter than that. :-) The version of the plugin is also considered. If Twenty Ten 1.5 is installed, only a translation designed for 1.5 (or earlier) would be installed. Try installing Twenty Ten 1.5 — you won't get a language pack installed. But if you update to Twenty Ten 1.6, you'll immediately get a pack.

[Chouby]

Replying to nacin:

Thankfully, it's a bit smarter than that. :-) The version of the plugin is also considered. If Twenty Ten 1.5 is installed, only a translation designed for 1.5 (or earlier) would be installed. Try installing Twenty Ten 1.5 — you won't get a language pack installed. But if you update to Twenty Ten 1.6, you'll immediately get a pack.

When I made the initial test, I just asked for a core update and did not notice that the theme was automatically updated to 1.6 (even if I did not want). Note that other (non-default) themes were not updated. So in some way you are right, the language pack matched the theme version. But not exactly the way you are describing. Just because in that case auto upgrade was acting

I tried with:

define('AUTOMATIC_UPDATER_DISABLED', true);

in wp-config.php but that did not avoid theme update as the same time as core.

I modified permissions of the theme directory to disallow theme update (but kept my languages directory writeable). And now as described above, I have unmatching themes and translations. I recognize that it's far-fetched and I got an 'update failed' message so I would not be surprised if something would be wrong late. But so far, I don't know how to avoid theme update at the same time as core to make the test as you described.

[dd32]

Replying to Chouby:

When I made the initial test, I just asked for a core update and did not notice that the theme was automatically updated to 1.6 (even if I did not want). Note that other (non-default) themes were not updated. So in some way you are right, the language pack matched the theme version. But not exactly the way you are describing.

What you're running into here, is that Development version upgrades (Automatic, or Manual) will always re-install default themes and plugins, This is because development users get trunk snapshots.

Stable Releases (ie. 3.6, and 3.7 final) are a bit different, a core upgrade does NOT upgrade a bundled Theme, so upgrading core doesn't result in the new theme (& language files) updating as well.

[nacin]

In 25784:

Hide language-specific warnings/labels on update-core.php when we are dealing with a point release partial build.

see #22704, #18200.

[dd32]

In 25806:

Language Packs: Many many fixes such as:

• Add a "Update Translations" stand-alone button to the updates page
• Shift Language feedback to before update process completion action links & limit the verbosity of output (name + success/errors)
• Simplify/combine the language update descriptive string to only include a plugin/theme name
• Properly handle cache clearing after language updates to prevent langs being repeditively updated
• Display a "All items up to date" string when there's nothing to do
• Reduce the 'Connection Information' from a <h2> to a <h3> to remove duplicate h2's and screen icons from update screens
• Fix the Direct filesystem method not being used for Language updates because WP_LANG_DIR doesn't exist (check it's parent for writable instead)

See #18200, #22704

[dd32]

In 25807:

Remove the no longer used Headerless_Upgrader_Skin class, it's been replaced by Language_Pack_Upgrader_Skin.
See #18200, [25806]

[nacin]

I suggest to add a select option for validators

All of this will be handled on the WordPress.org side in the coming weeks.

And with that, this ticket is done. New tickets please for any issues that crop up. Please let me know if I've missed anything spelled out here.

I'll note that one of dimadin's potential future improvements was "On unistallation of plugin/theme, associate language files should be deleted too." This is a mess, see 18200.3.diff​. Essentially, we're not going to act on this for now. (If someone wanted to create a ticket, they could.) If we wanted to, we could clear up unused language files on a future update, so it isn't a pressing issue right now.

[nacin]

In 25810:

Include translations in the update bubbles (as +1 if there are any). see #18200.

[dd32]

In 25812:

Fix a issue where the translation name isn't showing up in the Background Update emails, caused by a PHP Warning/missing variable in [25806]. See #18200

[dd32]

In 25818:

Allow the WordPress.org Language update API to disallow automated background updates for a specific language pack. See #18200

[JustinSainton]

[25818] introduced this line of code -

$upgrade = ! empty( $item->autoupdate );;

Assuming the double semi-colon was unintentional?

[dd32]

In 25822:

Remove a duplicate semicolon that snuck in. Props JustinSainton. See #18200

[nacin]

#15977 was marked as a duplicate.

[dd32]

#25044 was marked as a duplicate.

[nacin]

#26773 was marked as a duplicate.

[coolmann]

Is there a page somewhere that describes how to implement this functionality in a plugin?

[Ipstenu]

Replying to coolmann:

Is there a page somewhere that describes how to implement this functionality in a plugin?

​http://ottopress.com/2013/language-packs-101-prepwork/ ?

[coolmann]

Replying to Ipstenu:

​http://ottopress.com/2013/language-packs-101-prepwork/ ?

Thank you. However that page doesn't explain how/where to upload all the language files on WordPress.org or glotpress.

[dd32]

Replying to coolmann:

Is there a page somewhere that describes how to implement this functionality in a plugin?

Only WordPress.org-owned plugins can currently use this functionality.
It's due to be rolled out to all WordPress.org-hosted plugins when the functionality is available - We don't currently have a way to upload the translations or import them to a WordPress.org-hosted glotpress install.

Unfortunately it's taken much longer than expected for the WordPress.org infrastructure to get finished, there'll be an update here (and the plugins blog at least) when it's available.

[nacin]

Estimated timetable, FWIW, is before WordPress 3.9 is released in April.

[coolmann]

Thank you, guys. This will be a great functionality and it will save petabytes of data around the world! :)