WordPress.org

Make WordPress Core

Opened 9 years ago

Closed 5 years ago

Last modified 5 years ago

#15677 closed defect (bug) (fixed)

It is not possible to install multiple variants of the same language

Reported by: caesarsgrunt Owned by: ocean90
Milestone: 4.0 Priority: normal
Severity: normal Version: 3.1
Component: I18N Keywords: has-patch needs-testing
Focuses: Cc:

Description

Because of the way mu_dropdown_languages() and format_code_lang() work, it is impossible to have multiple variants of the same language installed. The list of languages returned by mu_dropdown_languages() only includes one variant of each language (the one with the country code which comes last when sorted alphabetically).

For example, if .mo files for es_ES and es_PE are both installed, then only es_PE will be shown.

Furthermore, whatever variant of the language is installed, the name will be the same, not indicating the correct language variant. In the example above, the language name will be "Spanish; Castillian", even for the es_PE localisation. This is highly misleading.

A final problem is that the langauge names are not localised themselves, with the exception of "English".

The cause of these problems is the fact that format_code_lang() ignores the country code and only considers the language code when choosing the name to return. Furthermore, it returns hardcoded language names rather than translated names.

Since mu_dropdown_languages() uses the return value of format_code_lang() as the key for its internal $output array, this means that later language variants overwrite earlier ones.

Attachments (23)

15677.diff (63.3 KB) - added by iandunn 6 years ago.
15677.2.diff (63.5 KB) - added by iandunn 6 years ago.
refreshed for 3.7
15677.3.diff (64.2 KB) - added by iandunn 6 years ago.
refresh
15677-lang-codes.diff (7.1 KB) - added by iandunn 6 years ago.
15677.4.diff (64.4 KB) - added by iandunn 6 years ago.
Fall back to iso code or slug if wp locale not present
15677-lang-codes.2.diff (7.5 KB) - added by iandunn 6 years ago.
ran against 4.diff
15677.patch (762 bytes) - added by ocean90 5 years ago.
15677.2.patch (20.0 KB) - added by ocean90 5 years ago.
15677.3.patch (21.1 KB) - added by ocean90 5 years ago.
show spinner while downloading language files
15677.4.patch (21.0 KB) - added by jesin 5 years ago.
Fix undefined notices and HTTP url fallback
15677.5.patch (22.4 KB) - added by ocean90 5 years ago.
15677.6.patch (26.4 KB) - added by ocean90 5 years ago.
Adds language chooser to network settings too
15677.7.patch (29.6 KB) - added by ocean90 5 years ago.
Ditch WLANG constant
15677.8.patch (26.8 KB) - added by DrewAPicture 5 years ago.
Docs adjustments
15677.9.patch (29.8 KB) - added by ocean90 5 years ago.
Merges .7 (based on .6) and .8
15677.10.patch (30.0 KB) - added by ocean90 5 years ago.
Add $wp_local_package to get_locale(). $wp_local_package can override WPLANG.
15677.11.patch (30.1 KB) - added by ocean90 5 years ago.
Don't set transients when WP_INSTALLING
15677.12.patch (26.9 KB) - added by ocean90 5 years ago.
15677.13.patch (27.0 KB) - added by ocean90 5 years ago.
In load_default_textdomain() load WP_LANG_DIR . "/admin-$locale.mo" when WP_INSTALLING
15677.14.patch (28.2 KB) - added by ocean90 5 years ago.
populate WPLANG option on upgrade again
15677.15.patch (29.9 KB) - added by ocean90 5 years ago.
15677.15.1.patch (29.7 KB) - added by ocean90 5 years ago.
Deutsch_Fortfahren.jpg (35.5 KB) - added by pavelevap 5 years ago.

Download all attachments as: .zip

Change History (74)

#1 follow-up: @nacin
9 years ago

  • Milestone changed from Awaiting Review to Future Release

I believe nikolay has suggested that there's a better alternative to mu_dropdown_languages() somewhere in GlotPress.

#2 in reply to: ↑ 1 @vanillalounge
8 years ago

Replying to nacin:

I believe nikolay has suggested that there's a better alternative to mu_dropdown_languages() somewhere in GlotPress.

nbachiyski suggested this in #13069

#3 @iandunn
6 years ago

  • Cc ian.dunn@… added

A workaround for this is to visit http://example.org/wp-admin/options.php and manually set the WPLANG value to pt_BR (or whatever). I'll work on a proper patch, but it'll probably be a few weeks before I have time.

Related #19760

#4 @iandunn
6 years ago

I've attached a first attempt at this, using locales.php from GlotPress to define the locales. I've got it running on WordCamp.org and it seems to be doing fine. The only issue is going to be backcompat, since the structure of the array has changed some.

e.g., $lang_codes['pt'] is now split into $lang_codes['pt_PT'] and $lang_codes['pt_BR']. There are some similar changes with other languages as well.

That could bite a plugin that's calling format_code_lang() directly, or using the lang_codes filter, since the input/output wouldn't always match what they're expecting anymore.

One inelegant solution would be to go through all the codes one by one, determine which ones have changed, and add duplicate entries for them in the array. That would be kind of lame, though, so I'm hoping someone has a better idea.

As far as updating the locales goes, it should be fairly easy to diff our version of the file against the original to bring in any updates.

I didn't internationalize the locale names since there's already a separate ticket for that (#19760), but it should be trivial to wrap them in __() in locales.php. Maybe even do it upstream to make maintenance easier.

If this is committed, it'd also fix #13069.

Last edited 6 years ago by iandunn (previous) (diff)

@iandunn
6 years ago

@iandunn
6 years ago

refreshed for 3.7

@iandunn
6 years ago

refresh

@iandunn
6 years ago

Fall back to iso code or slug if wp locale not present

@iandunn
6 years ago

ran against 4.diff

#5 @iandunn
6 years ago

  • Keywords has-patch added; needs-patch removed

15677-lang-codes.2.diff compares the $lang_codes array before and after the patch, to help assess the backcompat impact. A lot of the differences are just in ordering, but there's still a significant amount of changes from codes like pt to pt_BR and pt_PT.

The idea I had earlier about manually adding the codes that are missing (or changed) in GlotPress won't work, because then that language would show up in the dropdown multiple times.

Any ideas? Maybe we need to approach this from a different angle?

Or maybe there just isn't a good way to do this while maintaining backcompat? If that's the case, then maybe the best thing to do would be to just take this code and make it into a plugin that replaces the dropdown with a new one via the mu_dropdown_languages filter?

#6 @nacin
6 years ago

mu_dropdown_languages() is terrible. But there are actually deeper issues. There are two different pt_PT translations, one formal and one informal. Same goes for German. These situations will grow as more decide to do them.

I've been planning to pick this up starting in 3.9. Basically, locales.php from GlotPress is a major component, but we need to go further than that.

(Long-term goal is for language choosing to be able to just happen on Settings > General and for language pack files to then be downloaded into place.)

#7 @iandunn
6 years ago

Sounds good, thanks for the update.

This ticket was mentioned in IRC in #wordpress-dev by nacin. View the logs.


5 years ago

#10 @nacin
5 years ago

In 29007:

Introduce wp_dropdown_languages() and use it on general settings.

Early rough cut. Obviously not fully implemented.

see #15677.

This ticket was mentioned in IRC in #wordpress-dev by johnbillion. View the logs.


5 years ago

#12 @ocean90
5 years ago

  • Milestone changed from Future Release to 4.0

#13 @ocean90
5 years ago

In 29514:

Add a label to site language setting.

see #15677.

@ocean90
5 years ago

#14 @ocean90
5 years ago

15677.patch is an idea to quickly switch to a language which isn't even installed yet. On saving the translation packs should be downloaded automatically and when you get the screen back, it's already translated. What do you think?

Edit: Patch currently doesn't save the value, because sanitize_option() allows only one of get_available_languages().

Last edited 5 years ago by ocean90 (previous) (diff)

#15 @johnbillion
5 years ago

I like that idea. Here are a few brain dumps:

  • We should use two <optgroup>s instead of the separator made of dashes. One group for installed languages and one for available languages.
  • We should add a wrapper function for wp_get_available_translations_from_api() which uses a short term transient to avoid hitting the API every time the General Settings screen is viewed.
  • We should add a filter to allow the installation of new languages to be disabled.
  • Needs to handle API request failures.

#16 @ocean90
5 years ago

  • Keywords needs-patch added; has-patch removed

Good points working on a new patch.

This ticket was mentioned in IRC in #wordpress-dev by markoheijnen. View the logs.


5 years ago

#19 follow-up: @pavelevap
5 years ago

I am using cs_CZ locale (WPLANG). In Settings - General is in the first place en_US (should be currently used language). I tried to save it and swith to original English version and nothing happened.

#20 in reply to: ↑ 19 @ocean90
5 years ago

Replying to pavelevap:
Don't use the WPLANG constant, it's just a fallback now. When you set the setting to en_US the option will be empty and so WPLANG is used.

#21 @pavelevap
5 years ago

ocean90: OK, thank you. But what about older installations (updated from 3.9.x)? All will have WPLANG defined and they could switch language to English original (and back). There should be any Help tooltip available or en_US option unavailable for them.

#22 @knutsp
5 years ago

Core can detect WPLANG and en_US selection, and then display a message that WPLANG needs to be removed to make en_US work correctly.

Last edited 5 years ago by knutsp (previous) (diff)

@ocean90
5 years ago

#23 @ocean90
5 years ago

  • Keywords has-patch added; needs-patch removed

15677.2.patch is a current state, looks like this: https://cloudup.com/cxEoda9nxh6

#24 @netweb
5 years ago

Using 15677.2.patch:

Installed r29573 and installed language to Korean, changed to English (UK) and went to WordPress updates to download the English (UK) translations.

Edit: Scrap the 'date' issues, unless you change your timezone and date format of course this will occur:
https://cloudup.com/csCGjDOfHJL

On the dashboard Popular plugin and Install remain in Korean https://cloudup.com/cuU-QR27j0e

And the category title Uncategorized, remains in Korean, though I think this particular item will be wontfix and as it is 'content' end user should rename any 'created' content.

Last edited 5 years ago by netweb (previous) (diff)

@ocean90
5 years ago

show spinner while downloading language files

@jesin
5 years ago

Fix undefined notices and HTTP url fallback

#25 @jesin
5 years ago

Changes in patch 15677.4.patch:

  • Undefined variable $action in filters translations_api and translations_api_result changed to $type.
  • WP.org API http: URL assigned to the $http_url variable for fallback to work.
  • Variable $args in line 55 of translation-install.php changed to $options.

Bugs:

  • Switching back from any language to "English" (default) shows Settings saved. admin notice in the previous language. This is because the default "English" has an empty value attribute.
  • If there is problem communicating with WP.org and transients is empty the "Available Languages" list shows blank values except for English.

@ocean90
5 years ago

#26 @ocean90
5 years ago

  • Keywords needs-testing added

15677.5.patch:

  • Fixes bugs mentioned by jesin
  • translations_api():
    • $type = core: Pass WP version as $args['version'] to get core translations
    • $type = themes|plugins: Pass version as $args['version'] and theme/plugin slug as $args['slug'] to get theme/plugin translations
  • Only allow super admins to install languages when is_multisite()

ToDo:

  • Better handling of the default selected language. How should WPLANG constant and get_site_option('WPLANG') be handled, can we use get_locale() instead?

@ocean90
5 years ago

Adds language chooser to network settings too

@ocean90
5 years ago

Ditch WLANG constant

@DrewAPicture
5 years ago

Docs adjustments

#27 @ocean90
5 years ago

15677.7.patch:

  • Ditches WPLANG constant
  • On upgrade set WPLANG option based on the WPLANG constant
  • An empty WPLANG option means the site is in English
  • Adjust get_locale() so that an empty WPLANG option overrides the WPLANG constant

@ocean90
5 years ago

Merges .7 (based on .6) and .8

@ocean90
5 years ago

Add $wp_local_package to get_locale(). $wp_local_package can override WPLANG.

#28 @ocean90
5 years ago

There is currently one issue with wp-admin/upgrade.php: When WPLANG and $wp_local_package isn't set, get_locale() falls back to en_US because it doesn't load the options when WP_INSTALLING is set. This was added in [12900] (changed in [19059]) and [28774].

#29 @jesin
5 years ago

When installing WordPress with this patch, the "Information needed" screen shows the following two database errors:

WordPress database error: [Table 'wordpress.wp_options' doesn't exist]
INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES ('_site_transient_timeout_available_translations', '1408896686', 'yes') ON DUPLICATE KEY UPDATE `option_name` = VALUES(`option_name`), `option_value` = VALUES(`option_value`), `autoload` = VALUES(`autoload`)
WordPress database error: [Table 'wordpress.wp_options' doesn't exist]
INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES ('_site_transient_available_translations', 'a:41:{s:2:\"ar\";a:8:{s:8:\"language\";s:2:\"ar\";s:7:\"version\";s:9:\"4.0-alpha\";s:7:\"updated\";s:19:\"2014-01-01 00:00:00[...]

This is because wp-admin/install.php calls wp_get_available_translations() which tries to store the language data as transients.

@ocean90
5 years ago

Don't set transients when WP_INSTALLING

@ocean90
5 years ago

@ocean90
5 years ago

In load_default_textdomain() load WP_LANG_DIR . "/admin-$locale.mo" when WP_INSTALLING

#30 @ocean90
5 years ago

Patches .12 to .14 are going back to the main issue: Allow to install multiple variants of the same language, for example en_US, en_GB and en_AU, see also #28730.

To solve this we use wp_dropdown_languages() which will be populated by translations_api(). This will also fix #13069 and #19760 (probably wontfix).

15677.14.patch includes:

  • Add optional $locale argument to load_default_textdomain(), used in wp_switch_default_textdomain()
  • Remove !defined('WP_INSTALLING') from get_locale() for get_option( 'WPLANG' )
  • Add $wp_local_package to get_locale()
  • Allow an empty get_option( 'WPLANG' ) value to override WPLANG constant
  • Rename wp_install_download_language_pack() to wp_download_language_pack()
  • Rename wp_get_available_translations_from_api() to wp_get_available_translations(), API part extracted into translations_api()
  • Rename wp_install_load_language() to wp_switch_default_textdomain()
  • Remove define('WPLANG', ''); from wp-config-sample.php and populates WPLANG option on upgrade. That's not directly related to this issue, but makes some things easier to handle. Also I think 4.0 sounds like a good version number to get rid of this constant.

@ocean90
5 years ago

populate WPLANG option on upgrade again

@ocean90
5 years ago

@ocean90
5 years ago

#31 @ocean90
5 years ago

  • Owner set to ocean90
  • Resolution set to fixed
  • Status changed from new to closed

In 29630:

Language packs: No WPLANG anymore.

  • The WPLANG constant is no longer needed. Remove define('WPLANG', ); from wp-config-sample.php. Populate WPLANG option based on the WPLANG constant. When get_option('WPLANG') is an empty string it will override WPLANG.
  • Introduce translations_api() which is available to communicate with the translation API. Move translation install related functions to a new file.
  • Replace mu_dropdown_languages() with wp_dropdown_languages(). wp_dropdown_languages() is now populated by the translation API.
  • Remove wp_install_load_language() and allow load_default_textdomain() to switch a core translation.

fixes #13069, #15677, #19760, #28730, #29281.

#32 follow-up: @pavelevap
5 years ago

Nice, great work! Only small notice, "Note" is visible only when "English" language is selected. When I select "čeština" there is no "Note". Maybe it should be visible for all languages, it is a big change for localized versions and users should be notified that it is deprecated...

#33 in reply to: ↑ 32 @nacin
5 years ago

Replying to pavelevap:

Nice, great work! Only small notice, "Note" is visible only when "English" language is selected. When I select "čeština" there is no "Note". Maybe it should be visible for all languages, it is a big change for localized versions and users should be notified that it is deprecated...

This is by design, yep. Your WPLANG is set to pl_PL and so is your option, so there's no reason to nag you about it. This message is simply to cut down on support requests for people who change WPLANG and don't understand why it's not taking effect.

It does also issue a deprecated notice which will get picked up by Log Deprecated Notices, Debug Bar, etc.

#34 follow-ups: @pavelevap
5 years ago

OK, makes sense even if "Note" could be there in all cases.

I noticed another problem after 29630: During installation, "čeština" (please change it to "Čeština", my eyes are bleeding) was moved to the second place in dropdown (maybe locating my browser settings?), but there is still "Continue" and not localized version "Pokračovat" (maybe string changes?). Clicking on "Deutsch" works, there is localized "Fortfahren". But getting back to "čeština", there is still deutsch "Fortfahren" preserved. See attached screenshot.

#35 @ocean90
5 years ago

In 29633:

Language Form: The language option for $wp_local_package should get the same attributes as the other languages.

see #15677.

#36 in reply to: ↑ 34 @ocean90
5 years ago

Replying to pavelevap:

Thanks, fixed in [29633].

#37 in reply to: ↑ 34 @netweb
5 years ago

Replying to pavelevap:

During installation, "čeština" (please change it to "Čeština", my eyes are bleeding)...

Fixed https://glotpress.trac.wordpress.org/changeset/947

#38 follow-up: @pavelevap
5 years ago

ocean90: Nice, works well now!

netweb: OK, but I can still see only "čeština" during installation? Maybe it is cached somehow somewhere?

Anyway, WPLANG is deprecated now and how can I change language through administration?

Example (very common): Initial installation is created in English and users want to switch to Czech. It could be easily done changing WPLANG, but what is the recommended approach now? I did not find anything in Settings - General and WPLANG is deprecated... Or I can use Czech version and want to switch to Slovak (very similar language). How can I do it? Should I create new ticket?

#39 in reply to: ↑ 38 ; follow-up: @ocean90
5 years ago

Replying to pavelevap:

To switch the language you have to copy the translation files to wp-content/languages and then select it in the dropdown. So instead of touching the wp-config.php, you can now set it via the language dropdown.

Core will not install new languages. That was included in my patches until .13 but we decided do move this out of 4.0. I will create a new ticket for this feature and it will be a goal for 4.1.

#40 follow-up: @pavelevap
5 years ago

ocean90: OK, thank you. It is a little bit of regression, I guess. Now I could add only cs_CZ to wp-config.php and everything worked. In 4.0 I will have to download localization pack (it will be available on cs.wordpress.org?), extract it on local computer, find localization files, copy them through FTP client in the right place and in the end change language in dropdown. Too many steps for common users, I guess. Maybe do not deprecate WPLANG yet? I like the idea to not touching wp-config.php by users, because there are many possible problems (Fatal error, BOM, etc), but current workflow for changing language is an overkill. Please, do not deprecate WPLANG until it will be possible to switch languages easier...

#41 @markoheijnen
5 years ago

Ok wow. That decision is quite disappointed for me and I'm sure a lot of others because to me that was one of the main things I expected from 4.0.

#42 in reply to: ↑ 40 @ocean90
5 years ago

Replying to pavelevap:

Just setting the WPLANG constant doesn't provide you the language files in 3.9, so there is no difference.

#43 @pavelevap
5 years ago

ocean90: That is not true, simply setting WPLANG and going to Dashboard - Updates to automatically install localized version works well in 3.9.x. So it is a regression and should be solved somehow... And it was also planned for 4.0.

Another related ticket (other problem with dropdown functionality): #29297

#44 @johnbillion
5 years ago

If we're installing new language files during installation, what's stopping us installing new language files when the WPLANG option is changed in the admin area? Anything I can help with?

#45 @pavelevap
5 years ago

I am not sure about the best solution, but now it is a mess. WPLANG is deprecated now, users have to manually copy language files through FTP, because adding languages through administration does not work. And when they copy only .mo files, they can switch to new language, but automatic language packs will not work (#29297).

So, generally we are telling to users: In 3.9.x you could simply do it with WPLANG, in 4.0 you have to copy files (and you have to learn what to do and how) and everything will be changed in 4.1 once again? Not good, long Codex page needed...

#46 in reply to: ↑ 39 @ocean90
5 years ago

Replying to ocean90:

Core will not install new languages. That was included in my patches until .13 but we decided do move this out of 4.0. I will create a new ticket for this feature and it will be a goal for 4.1.

See #29395.

#47 @pavelevap
5 years ago

ocean90: OK, but you should reconsider including this feature into 4.0. You are presenting really bad user experience for adding languages in 4.0. Better postponing release for some days than annoying users with current functionality, I guess...

#48 @pavelevap
5 years ago

Is it OK to have 2 strings for 1 sentence? "Note:" and "The %s constant in your %s file is no longer needed."? What about RTL languages?

#49 @pavelevap
5 years ago

Current behaviour is regression and there are other posts about this issue, see: http://make.wordpress.org/polyglots/2014/08/29/let-me-share-some-issues-about-4-0-language/

It will be really postponed and not included in 4.0?

#50 @knutsp
5 years ago

I agree that current behaviour, as described, is a regression. I 3.9 you can just change the locale in wp-config.php and update WordPress to a new version, as 3.9.2 nb_NO to 3.9.2 sv_SE.

Over the years I have advised several clients that somehow got the wrong language version installed, to correct this by just editing wp-config.php. So, if this happens again and the current functionality will be what's released, I will have a hard time advising these clients to help themselves through a simple file edit. We are back to the situation many years ago, when, in these cases, you had to get the latest translation files downloaded and put them in the right place, and change WPLANG.

So this either isn't ready to go, or a decision must be made to openly accept regression and make apologies for that. 4.0 should be great when it comes to localization.

#51 @ocean90
5 years ago

#28736 was marked as a duplicate.

Note: See TracTickets for help on using tickets.