Allow language to be chosen during initial install

[nacin]

The first step installing WordPress should be to choose a language. The rest of the install process would then be in that language.

As outlined in ​http://make.wordpress.org/core/2014/05/21/internationalization-goals-for-4-0/, we'd like something like this:

Multiple workflows

One issue that came up in #26879 (“friendlier welcome when installing WordPress”) is that setup-config.php is often not the initial entry point. A user installing WordPress through a hosting control panel is likely to be dumped onto the install screen (if not the dashboard directly).

Thus, we need to keep in mind that either screen might be the “intro” screen. This introduces technical challenges: If the user’s first step is setup-config.php, we don’t actually have WordPress fully loaded at this point, which makes actually installing a language pack more difficult. The install step has WordPress loaded in full, just without database tables. We should look into making setup-config.php load “all” of WordPress to make these environments easier to code.

How to approach this

There are two different ways to approach this: download language files immediately upon selection, or bundle barebones language files of the install screens for all supported languages. The latter is not the easiest thing to do (due to error messages and such, strings can come from all over) and also adds a delay to the adoption of new languages. The former would mean that we would want to pull a languages list from WordPress.org. (If we cannot reach WordPress.org, we would have no way of downloading the complete language pack, so this is not a big deal.) It’s still a bit of a challenge due to the split workflows problem.

The plan is to download language files immediately upon selection.

Language suggestions

WordPress.org already recommends a language based on the browser’s settings (Accept-Language header) as well as using a more rough lookup based on IP address blocks. (HTML5 location could be used, but unless we’re also going to use that to set the user’s timezone, it seems excessive for a “choose your language” for the moment, especially since location is not as preferred anyway.)

Both would require the client to hit WordPress.org via JavaScript, versus a server HTTP request. We can do a server-side HTTP request to generate the list, then a client-side request to float recommended ones to the top. It’s possible to do it in two steps: the Accept-Language mapping can be done locally, while the IP-to-location table on WordPress.org has 2.9 million entries and would require a round-trip. Of course, if a user has downloaded a localized package directly from a local WordPress.org site, that would be the top recommendation.

Locale variants

Note that by language or locale we also must consider other translation variants, as there may be a language like Portuguese available in multiple locales (Brazil, Portugal) and further broken down by formal and informal variants. Each of these would be listed as their own “language.” c.f. #28303

API

​https://api.wordpress.org/translations/core/1.0/?version=4.0-alpha is functional.

[nacin]

28577.diff​ is a first pass.

[nacin]

In 28774:

Allow a language to be chosen before installing WordPress. First pass.

• Checks WordPress.org for available languages.
• In get_locale(), starts using the WPLANG option that has existed in multisite since the MU days.
• Adds new argument to wp_install() for setting WPLANG.

see #28577.

[johnbillion]

I know it's early days, but this currently causes the "Allow search engines to index this site" checkbox to be unchecked by default because it's sending POST data to install.php without a value for blog_public.

[nacin]

Replying to johnbillion:

I know it's early days, but this currently causes the "Allow search engines to index this site" checkbox to be unchecked by default because it's sending POST data to install.php without a value for blog_public.

Great catch, I had definitely missed that. (Doesn't help when the screen was mostly in languages I did not understand during testing.)

[nacin]

In 28779:

Fix variable usage in new language install choices. see #28577.

[nacin]

In 28780:

Narrower screen when choosing a language. see #28577.

[nacin]

The to-do list here is not short. If you want to help out, there's a list at the bottom. Main things to do: better error & filesystem handling, setup-config.php, locale variants, improved localization, language suggestions. The details:

On errors:

• Before showing the language chooser, pre-flight whether we can actually download and install language files using 'direct' file access. Thinking we ignore FTP?
• Gracefully fail if we failed to install the target language. Right now it'll simply load everything in English and not end up setting WPLANG. While this is graceful, it is also silent. Desired?

On implementation:

• Bring this to setup-config.php somehow. The wp_die() in wp-load.php is troublesome; we'll probably need to come up with an all-new workflow and code flow.
• Locale variants need to be designed and implemented.

On localization:

• Rather than using the native name, use "Use English as the main language" (but translated into the language). This will be translated on translate.wordpress.org but actually served from the API. Also, rather than an arrow, we can use a translation of "Continue" or "Let's go!" or "Get started" (again, served from the API) for the button.
• Certain languages are best displayed in certain ways. We need to investigate these. For example, Hebrew should be in RTL including Arabic should be in Tahoma. Of course, this is pretty easy to configure via the API. Might these need to be also aligned right? There's also a language in central Asia that requires its own typeface, if I recall correctly. We will probably need to abandon this multi-line select box in favor of something more robust in order to improve styling.

The order of language suggestions:

• If we detect a language based on your browser, this should be first. If we detect multiple, we should list them in the order specified.
• If we are able to guess one or more language suggestions based on your IP address, these should come next.
• If you downloaded a particular language's ZIP package, this language should be listed next. (It's possible this should actually be the very first one.)
• English (US) should then be listed.
• All other languages should be listed in some kind of alphabetic order. Currently the API is ordered by ISO per an initial suggestion from Zé (vanillalounge), but this means languages like Spanish (Spain) and Spanish (Chile) are not listed anywhere near each other. We could benefit from some kind of natural sort, even if this is done manually (based on the order from GlotPress, for example).

If you want to help, the best things to help with are:

• The 'errors' stuff, which should be straightforward-ish but will require some familiarity (or desire to gain familiarity) with WP_Upgrader and friends.
• The setup-config.php stuff. This is probably gonna be a sizable refactor; may require a few attempts / false-starts. (Show your work; it helps to see what others have tried.)
• Research how to best display and sort native names. How do others do it? (OS X, Ubuntu, Drupal, Joomla...)

I'll be working on the more complex tasks that require heavy .org integration:

• Suggestions
• Locale variants
• Leveraging string translations via the API

[johnbillion]

What should happen if WPLANG is already set to a non-default value when running the installer? Should we skip the language selection, or should it still be shown but with that language pre-selected?

Subsequently, will we see issues if the WPLANG constant is set to one language, but the site gets installed with another so the get_option('WPLANG') ends up being different? (Thinking of plugins that use WPLANG directly instead of get_option('WPLANG').

[johnbillion]

Here's a dump of screenshots of the language switcher from various installation screens:

Drupal 7.28

Joomla 3.3

RedHat Enterprise 5

Ubuntu Desktop

Windows 7

Windows 8

Windows Essentials

Some Observations

OS X and Windows 7 use a sentence for each language ("Use %s as the main language" and "My language is %s" respectively). All other language selectors I've seen just use the language name on its own. Sometimes this is just the native name, sometimes it's the English name alongside the native name.

A drop down list is not obviously a language selector if you can't read English. A list that's at least partially visible is much more clearly a language switcher.

The Joomla! install process has added complexity because it needs to force a reload of the page if you change language.

The Windows Essentials language switcher splits the list into installed and available languages. Although we should avoid this in WordPress, we could list the currently installed languages (if any) at the top of the list.

Below is a screenshot of the language switcher on Google. It demonstrates how long the list of languages could potentially get. The less common languages are hidden by default behind a 'show more' link.

[johnbillion]

I think we should go with an unordered list containing radio buttons. The list items would float into columns as necessary.

IMHO the English name of a language is not of much use. I think just the native name should be used in the list.

[johnbillion]

We should add a loading spinner next to the submit button on the language selection screen. It could potentially be slow to download translations on a slow connection.

[nacin]

Replying to johnbillion:

What should happen if WPLANG is already set to a non-default value when running the installer? Should we skip the language selection, or should it still be shown but with that language pre-selected?

Shown pre-selected. This is part of the suggestions work.

Subsequently, will we see issues if the WPLANG constant is set to one language, but the site gets installed with another so the get_option('WPLANG') ends up being different? (Thinking of plugins that use WPLANG directly instead of get_option('WPLANG').

Plugins using should be using get_locale(). Neither of these are valid to use.

[johnbillion]

Replying to nacin:

Replying to johnbillion:

Subsequently, will we see issues if the WPLANG constant is set to one language, but the site gets installed with another so the get_option('WPLANG') ends up being different? (Thinking of plugins that use WPLANG directly instead of get_option('WPLANG').

Plugins using should be using get_locale(). Neither of these are valid to use.

Unfortunately a grep of the plugin directory shows just over 400 plugins using WPLANG in one form or another.

[sonjanyc]

I've worked on some mockups for the language selection screen a while ago: ​http://make.wordpress.org/core/2014/05/21/internationalization-goals-for-4-0/#comment-15796

Looks like @nacin prefers the translated "Use english as language" phrase rather than just a dropdown of all languages. Which is fine with me, just thought it might be easier to find your language if it's just the name of the language in an alphabetical list. But either approach works.

Are there other screens I can work on, or error states? Let me know.

[azaozz]

Replying to nacin:

On errors:

• Before showing the language chooser, pre-flight whether we can actually download and install language files using 'direct' file access. Thinking we ignore FTP?

Been thinking what we can do in these cases. One possibility is translating from JS (similar to how TinyMCE does it). We do a JSONP request that returns an translation object { 'English string': 'Translated string', ... }. Then on DOM ready we walk the DOM and replace (may wrap all strings in something like <span class="wp-translate"> to make that a bit easier/faster). This may make the page flicker a bit.

Another (better?) possibility would be to get an installation-only .mo file from the WordPress.org API on each page load and use it directly (keep it in memory). Yes, it's expensive and will slow down the 3-4 page loads while installing, but will work in all cases.

This can also be used on the FTP setup screen (right after we are up and running, or before we start?) so the user can install the language pack.

[netweb]

Replying to nacin:

API

​https://api.wordpress.org/translations/core/1.0/?version=4.0-alpha is functional.

Nacin, could you please add English (Australia) to the API ;)

[jorbin]

Above is an early and incomplete update to the HTML. It uses radios and labels that each have unique classes (currently based on the language, but it could be based on something else from the API) so that they can be styled differently if need be.

It's missing (at least) a few things:

• The fieldset isn't styled based on focus.
• The fieldset doesn't properly scroll when navigating via the keyboard
• The continue input needs a label. We may want to consider actual text here rather than the »

[johnbillion]

28577.diff​ is looking good. I'm going to do a few tweaks shortly.

[johnbillion]

In 28982:

Avoid a warning if the translations API call fails on step 1. Remove assignment inside an if statement. See #28577.

[johnbillion]

In 28983:

First pass at switching to radio inputs for the install screen language selector. Needs some more work. See #28577. Props jorbin

[johnbillion]

In 28984:

If we already have a value for WPLANG when installing, insert this language at the top of the language selector and pre-select it. See #28577

[michalzuber]

I get Fatal error now trying the language selection during the install. #28744

[michalzuber]

Why is the styling of the inputs needed? In select I just type the starting characters of the desired language and I don't need to scroll that much. Here I don't know if english_name or native_name should be used due the key typing.

[nacin]

In 29006:

Bring the language chooser to setup-config.php.

see #28577.

[nacin]

As of [29006], either setup-config.php or install.php work well as entry points. A language chosen during setup-config.php gets passed through to install.php via a simple GET parameter.

I think I'd like to drop a cookie and then read from that, but this current implementation allows for shortcuts like install.php?language=de_DE, which is not bad. We'd still have to thus account for other request variables. (In WordPress, $_REQUEST is GP, not GPC.)

I moved the assignment back into the if statement (see [28982]) as it improves the branching. I added parentheses to make it a bit more obvious an assignment is there.

More to do here:

• Suggestions — johnbillion said he has some solid Accept-Language code. So do I, but I wouldn't call it solid. Waiting to see his code.
• Language variants — I have an idea here.
• Leveraging string translations via the API — working on this. (jorbin pointed out this is necessary for accessibility.)
• Improved update/error handling — up for grabs.

[michalzuber]

Parameter from function is not an object

[michalzuber]

The upgrader doesn't download the language pack even with write perm. Some $GLOBALS and $upgrader output ​http://pastebin.com/kJruEs2S Couldn't figure it out why. The passed $language was

stdClass Object
(
    [language] => sk_SK
    [version] => 4.0-alpha
    [updated] => 2014-01-01 00:00:00
    [english_name] => Slovak
    [native_name] => Slovenčina
    [package] => https://downloads.wordpress.org/translation/core/4.0-alpha/sk_SK.zip
    [type] => core
)

[jorbin]

The above patch adds in focus/blur styling to the language chooser. These may not be the final colors, but it is something to get us started there.

Next up from me is going to be keyboard scrolling of the inside box when navigating with a keyboard.

Following that, keyboard based searching to jump to the language of your choice.

[michalzuber]

Renamed wp_get_available_translations() to wp_get_available_languages(), removed unused $options from upgrade.php, started error notification

[johnbillion]

In 29016:

Correct the usage of the variable passed to wp_install_download_language_pack() and kill an unused local variable. Props michalzuber. Also avoid using the $_REQUEST superglobal in this function. See #28577.

[johnbillion]

28577.4.patch​ adds a first pass at browser accepted language detection, and pushes these languages to the top of the language list, below 'English (United States)' and below the WPLANG language, if defined.

[nacin]

In 29018:

Language on install: Docs and naming cleanups. see #28577.

[johnbillion]

In 29020:

Focus and blur styling (via JS) for the install screen language selector. Props jorbin. See #28577

[johnbillion]

In 29021:

Narrower layout and loading spinner for the install screen language selector. See #28577

[johnbillion]

jorbin: The hover/focus styles are looking good. I wonder whether the blurred selected state should be blue though (instead of grey), to liven it up a bit when you first land on this screen.

Also, is anyone else seeing odd jumping behaviour when you select a language which you've needed to scroll down the list to get to? It seems to be related to the presence of the screen-reader-input class on the radio inputs. If that's removed, the jumping doesn't occur. Only tested so far in Chrome.

[netweb]

Replying to johnbillion:

Also, is anyone else seeing odd jumping behaviour when you select a language which you've needed to scroll down the list to get to? It seems to be related to the presence of the screen-reader-input class on the radio inputs. If that's removed, the jumping doesn't occur. Only tested so far in Chrome.

I don't see any odd 'jumping behaviour' in Firefox 30, Chrome 38 or IE11.

What I do see though is as I cursor down using arrow keys the scrollbar does not follow the selections as I continue to cursor down so the current selected language is 'off screen'.

e.g Scrolling down to 'Magyar' this is what I see, the scrollbar should follow (Same behaviour in same browsers above)

[michalzuber]

Added scrolling with keys to fieldset

[michalzuber]

Experiencing the odd jumping in Chrome, but not in FF. Tried to debug ​http://youtu.be/ONlOrA1v6YU but no success :/

[nacin]

In 29058:

Install: Don't ask for the user's language a second time if they chose English originally.

see #28577.

[nacin]

In 29059:

Unload the current default textdomain in wp_install_load_language().

see #28577.

[nacin]

[29058] fixes the issues reported variously where English would cause the second language chooser. I had no idea how to reproduce this until I saw ryan's post on make/flow (​http://make.wordpress.org/flow/2014/07/09/scratch-install-with-setup-config-macnchrome/).

[29059] fixes an issue when the install.php screen would listen to WPLANG (such as one set up in a wp-config-sample.php file in a non-English zip download) instead of the chosen language. I didn't completely debug it (it doesn't make complete sense to me as $mo->merge_with() should have meant the later language won), but this fixed the issue and was good enough for beta 1. More investigation to follow.

[YagoStecher]

Where should I report a small bug on the localizaton of this very installation screen? Spanish message is incomplete in the confirmation screen of "step 2". (after selecting language, vefore starting the installation)

[pavelevap]

Nice, but I still do not understand localization workflow. WordPress 4.0 will be released and translation will not be ready. So users will receive 3.9 localization? And when 4.0 localization is ready, there will be some automatic language update? And how can translators mark translation as complete to allow distribution?

[netweb]

Replying to YagoStecher:

Where should I report a small bug on the localizaton of this very installation screen? Spanish message is incomplete in the confirmation screen of "step 2". (after selecting language, vefore starting the installation)

I think you are referring to this string?

We're going to use this information to create a wp-config.php file. (src: setup-config.php#L126)

This actually affects all locales, the string isn't available in GlotPress for translation.

[YagoStecher]

Replying to netweb:

Replying to YagoStecher:

Where should I report a small bug on the localizaton of this very installation screen? Spanish message is incomplete in the confirmation screen of "step 2". (after selecting language, vefore starting the installation)

I think you are referring to this string?

We're going to use this information to create a wp-config.php file. (src: setup-config.php#L126)

This actually affects all locales, the string isn't available in GlotPress for translation.

Actuallly it's this bit

If you are ready, time now to…(setup-config.php#L318)
...but I can see it's incomplete in the english version, too.

This is the screencap ​http://prntscr.com/41bc67

[netweb]

Replying to YagoStecher:

Actuallly it's this bit

If you are ready, time now to…(setup-config.php#L318)

Cool, you can update that translation at the following URL (login using your wordpress.org user/pass)

​http://translate.wordpress.org/projects/wp/dev/admin/es/default?filters[status]=either&filters[original_id]=55132&filters[translation_id]=2499767

After you have updated it, send the Spanish translation team a quick note ​http://es.wordpress.org/contact/

[pavelevap]

Where can be "čeština" changed to "Čeština"?

[johnbillion]

In 29260:

Add a lang attribute to each language listed on the language selector screen during installation. See #28858, #28577.

[johnbillion]

I'm looking into the possibility of switching back to a select input (with a size attribute) rather than our current list of radio buttons.

The current list uses a fair amount of CSS and JavaScript which is just there to get it to look and behave like a native select input. In addition, we're still seeing odd jumping behaviour in Chrome when selecting a language after scrolling down the list, the list doesn't scroll when using the arrow keys to navigate, and typing a letter on the keyboard doesn't jump to a language beginning with that letter.

If we switch back to a select input, we avoid all these issues. As a bonus, we get the native select UI on Android/iOS devices.

Replying to nacin:

Certain languages are best displayed in certain ways. We need to investigate these. For example, Hebrew should be in RTL including Arabic should be in Tahoma. Of course, this is pretty easy to configure via the API. Might these need to be also aligned right? There's also a language in central Asia that requires its own typeface, if I recall correctly. We will probably need to abandon this multi-line select box in favor of something more robust in order to improve styling.

On ​Google's language preference screen it lists over 150 language names, all of which are shown in Arial and left-to-right.

After some discussion with nacin, #19950 was mentioned. The Uyghur language requires its own font for general use, but the language name on its own displays correctly in Arial. Uyghur can be found on Google's language preference screen as ئۇيغۇرچە.

[johnbillion]

28577.6.patch​ switches the language chooser back to a select element (with a size attribute). Feedback welcome.

[michalzuber]

Replying to johnbillion:

28577.6.patch​ switches the language chooser back to a select element (with a size attribute). Feedback welcome.

Definitely agree with johnbillion. With <select> searching by letter keypresses would also work without JS hacking, ​KISS.

PS: Could we also change that button value "»" to "Continue in selected language", "Continue", "Use language" ​http://i.imgur.com/ZS2FBIM.png ?

[DrewAPicture]

+1 for moving back to using <select> with 28577.6.patch​. It seems like we get a lot more compatibility "for free". Plus it's just cleaner overall.

[charlestonsw]

This is how the language selector is appearing in Firefox ESR 24.7.0 (latest rev on CentOS 6.5).

What fixed it on my box?

Adding a wrapper div around the fieldset and setting THAT CSS rule to overflow: auto; worked.

in wp-admin/includes/upgrade.php @ wp_install_language_form() add: echo "<div id='language_selector'>"; just before the echo "<fieldset>\n" and close it out with echo "</div>" after the closing fieldset in the same function.

CSS rule added to install.css

.language-chooser #language_selector {
    overflow: auto;
}

CSS rule modified in install.css

.language-chooser fieldset {
   margin: 1px 1px 0;
}

What did NOT work

The problem looks like a FF bug but there are no newer patches for the ESR version at this point.

The Mozilla-specific CSS hack to hides fieldset overflow but does not render the scrollbar.

/wp-admin/css/install.min.css?ver=4.0-beta2-20140720

.language-chooser fieldset {
   overflow: -moz-hidden-unscrollable; 
}

overflow: auto;
overflow: scroll;
overflow-y: auto;
overflow-y: scroll;

And a myriad of other options are not helpful.

HTH.

• Lance

[johnbillion]

Thanks Lance. At this point it looks most likely that we're going to revert to a select element to keep it simple (see 28577.6.patch​).

[charlestonsw]

@johnbillion - I think select is a wise choice. It has better cross-platform support even if it is a lot harder to craft a nice UX around it.

At least the "div patch" is noted in case others run into fieldset styling issues in the future.

<tangent>It is 2014... why is HTML + CSS styling still such a pain?!?!</tangent>

[azaozz]

Replying to johnbillion:

28577.6.patch looks pretty good. Couple of small nitpicks: you don't need to add selected="selected" on the first option, it is selected by default. The different browsers would let you style a <select> a bit differently, but generally many styles are ignored on <option>, perhaps the font-size can be 18px to make it a bit more prominent. And of course the size (visible rows) can be changed from JS, perhaps useful for mobile.

[nacin]

In 29372:

Language chooser imporvements.

• Use a translated 'Continue' string.
• Go back to using a select element.
• Only show a spinner when we're installing a language.

see #28577.

[nacin]

Also [29374]

[jorbin]

The color contrast for selected elements is 4.473867036212602:1 . WCAG 2.0 AA Guidelines suggest a minimum of 4.5:1 for small text (less than 18 point). 1 pixel is roughly 1 point, so if we bump the font up 2 px, we will meet that guideline. Additionally, I think the slightly larger font is easier to read.

[pavelevap]

Where can be language name "čeština" changed to "Čeština"? Should I create new ticket?

[markoheijnen]

Hey Pavelevap, There is a ticket about that on the GlotPress trac. See ​https://glotpress.trac.wordpress.org/ticket/342.

[netweb]

Replying to netweb:

We're going to use this information to create a wp-config.php file. (src: setup-config.php#L126)

Have created a new ticket for this issue and another string in #29233

[johnbillion]

In 29516:

Avoid a PHP notice if you manually navigate to /wp-admin/install.php?step=2. See #28577.

[Blaine Moore]

readme file

[Blaine Moore]

Site Language screenshot in general settings

[Blaine Moore]

At WordCamp Maine Contributor's Day I tested installing the latest nightly, and saw a few issues which I didn't see mentioned here and couldn't locate separate tickets for.

1. After installing (on wp-admin/install.php?language=xx_XX) there is a link at the top to the readme.html file; the readme.html file is in English. Is there any way for it to download a translated version if a language other than English is chosen and display that readme file instead of the default one?

2. After installation is completed and visiting Settings -> General (on wp-admin/options-general.php) there is an option for Site Language, which lists all installed languages. However, instead of using the language name in the drop down, it uses the locale value, which may be confusing to an end user that may not know his specific regional code.

[ocean90]

Replying to Blaine Moore:

1. After installing (on wp-admin/install.php?language=xx_XX) there is a link at the top to the readme.html file; the readme.html file is in English. Is there any way for it to download a translated version if a language other than English is chosen and display that readme file instead of the default one?

Yeah, it's planned to download the localized readme.html too.

2. After installation is completed and visiting Settings -> General (on wp-admin/options-general.php) there is an option for Site Language, which lists all installed languages.

This is WIP and will be improved, see #15677.

[helen]

Comments here are wandering a bit and hard to parse - let's close this as fixed (the selector is in place on the installation screen) and open new tickets for individual issues.

[johnbillion]

Follow-ups: #15677, #29290, and ​https://glotpress.trac.wordpress.org/ticket/342