Check and recommend intl extension in Site Health

[zodiac1978]

First reported here: ​https://github.com/WordPress/health-check/issues/333

I suggest to check for the existence of the intl PHP extension with the status of a recommendation.

This extension is necessary to use the Normalizer function which would solve many problems with internationalization issues for special characters:
​https://www.php.net/manual/en/intl.requirements.php

Unfortunately this extension is not included per default on every PHP/hoster.

Related issues: #30130, #35951, #24661, #47763

Additional info from the Github issue:

intl has hard dependency on icu system libraries, which will consume over 30 Mb diskspace on smaller (VPS/embedded) systems. Each loaded library claims some memory and initialization time overhead.

Bigger distributions can link libicu to libxml2 library and therefore it can be loaded by libxml2 linked extensions, but this dependency is optional

The site health check for this extension could be easily added like shown here:
​https://gist.github.com/Zodiac1978/ce9db99becd3507ec8d8207c583e947b

This was discussed in the hosting team with some additional comments:
​https://make.wordpress.org/hosting/?s=%22intl+extension%22

Especially this feedback from @swissspidy is interesting:

in the case of the INTL extension, when a server is running PHP 7.2 but uses a super outdated version of the extension which doesn’t include the functions and constants one would expect. So any site health check for extensions should also check against these things.

Therefore there are two ways to check here. We could check for the extension itself or we could check for the function(s) we need in core, like the Normalize function which could help resolve the bug tickets mentioned above.

I would recommend the latter.

Any input on how to get forward here from the Hosting team and the Site Health people is much appreciated!

[Clorith]

Is the Normalizer class still maintained and going to be a thing moving forward?

A quick look at the ​PHP documentation shows it's available in PHP 5.3 and PHP 7, but not PHP 8, but then rather as an optional install via the intl PECL package.

The latter makes it seem like a non-starter based on the linked Hosting Team discussion on criteria for recommendations, but I'm open to it if the Hosting Team feels strongly about this as well, as the ones most likely to get requests for this.

[JavierCasares]

In the "intl" there are several functions.... some summary of use:

+ ​Locale

• WooCommerce
• Yoast SEO
• Jetpack
• Contact Form 7
• Wordfence Security
• ...

+ ​Normalizer

• Contact Form by WPForms
• Facebook for WooCommerce
• BackWPup
• File Manager
• Beaver Builder
• ...

+ ​MessageFormatter

• UpdraftPlus
• Beaver Builder
• Toolset Types
• Post SMTP Mailer
• Mollie Payments for WooCommerce
• ...

+ ​Collator

• WooCommerce

+ ​NumberFormatter

• WP RSS Aggregator
• Matomo Analytics
• Unite Gallery Lite
• Amelia

+ ​IntlTimeZone

• MailPoet
• WP RSS Aggregator

+ ​IntlDateFormatter

• MailPoet
• WP RSS Aggregator
• CAOS

+ ​Spoofchecker

• MailPoet

+ ​Transliterator

• GiveWP

+ ​UConverter

• WPeMatico RSS Feed Fetcher
• CyberSEO Lite
• WordPress RSS Feed Retriever

+ ​IntlChar

• Simply Static
• Smart Grid-Layout Design for Contact Form 7

[zodiac1978]

Replying to Clorith:

Is the Normalizer class still maintained and going to be a thing moving forward?

A quick look at the ​PHP documentation shows it's available in PHP 5.3 and PHP 7, but not PHP 8, but then rather as an optional install via the intl PECL package.

The latter makes it seem like a non-starter based on the linked Hosting Team discussion on criteria for recommendations, but I'm open to it if the Hosting Team feels strongly about this as well, as the ones most likely to get requests for this.

As I understand it, it was developed in the past as PECL package:
​https://pecl.php.net/package/intl

But nowadays it is an official part of PHP:
​https://www.php.net/manual/en/intro.intl.php

The underlying code was in both cases the maintained ICU library:
​http://site.icu-project.org/home

[JavierCasares]

After the last session and after several meetings, the Hosting team has considered that the installation of the PHP-intl extension should be recommended (but not mandatory).

​https://make.wordpress.org/hosting/2021/05/06/hosting-team-meeting-summary-2021-05-05/

[jrf]

A quick look at the PHP documentation shows it's available in PHP 5.3 and PHP 7, but not PHP 8, but then rather as an optional install via the intl PECL package.

@Clorith You may have looked before the manual was updated, but yes, the Intl extension is available in PHP 8 and it also states as much in the manual (at the time of writing):

(PHP 5 >= 5.3.0, PHP 7, PHP 8, PECL intl >= 1.0.0)

More relevant is that PHP ​needs to be build with `--enable-intl` for it to be available, so a functionality check (function_exists/class_exists/defined etc) is always needed before any of the Intl functionality can be used..

[jrf]

Oh and just FYI, the non-availability could - for a limited set of functionality - be worked around by using the ​Symfony polyfills for the Intl extension.

[zodiac1978]

@jrf Thank you for the details about the extension and why it is not always available!

The polyfill could be useful for all the mentioned tickets, I think:

Related issues: #30130, #35951, #24661, #47763

Otherwise there is a chicken/egg-probleme here. Core does not use it, because it is not always available. Hoster do not implement it, because it is not required/used in core.

Hopefully this recommendation from the hosting team and the polyfill could be a starting point for these tickets. This would be a huge win for all non English speaking people using WordPress.

[jrf]

Replying to JavierCasares:

In the "intl" there are several functions.... some summary of use:

I've just had a quick look through the search results and feel like I need to add a word of warning on the value of these as there appear to be quite some false positives due to the searches not being precise enough.

That by no means invalidates the drive to get people to activate the extension as I totally agree the Intl extension really does add significant value if we could use it.

It just means that the "plugin use" information as part of the making the case for recommending Intl is flawed.

+ ​Locale

Checked the complete first page and only found 8 legitimate uses out of 100 entries. Everything else was false positive, including the whole top five listed above. Most JS or PO files.

+ ​Normalizer

Verified first x entries and found mostly correct. About 25% false positives from the Guzzle README.md file, some more false positives for namespaced classes also called Normalizer`. Rest mostly coming from the Symfony Iconv/Intl polyfills.

+ ​MessageFormatter

Verified first x entries and nearly all are false positives. Mostly use in comments or the Guzzle UPGRADING.md file. Only found two legitimate uses from the Symfony Intl polyfill.

+ ​Collator

Verified first x entries and found mostly correct. About half coming from Symfony Intl Polyfill, the other half from a "free world map" function. Saw at least one false positive in a JS file.

+ ​NumberFormatter

Verified first x entries and found mostly correct. Saw at least one false positive in a JS file.

+ ​IntlTimeZone

Verified first x entries and found correct. About half coming from Twig, the other half from Symfony Validator.

+ ​IntlDateFormatter

Verified first x entries and found correct. About half coming from Twig.

+ ​Spoofchecker

Verified and found correct. Most coming from the Egulias\EmailValidator package.

+ ​Transliterator

Some false positives for use in comments or translatable strings. Some real use.

+ ​UConverter

Verified and found correct. Mostly found in a SimplePie code (not the WP Core version).

+ ​IntlChar

Verified and found correct.

[JavierCasares]

Replying to jrf:

Yes, we checked that before doing the Hosting Meeting. Also, we saw that the PHPMailer inside the core has some "intl" functions (with an alternative IF in all the cases).

The thing is Core / Plugins doesn't want to include some "intl" functions because a lot of hosters doesn't have the extension, so we are recommending to start installing it, so it will be available to use.

We will add it to the Hosting Handbook soon.

Sorry for the first confusion, but we checked it later in our meetings.

[jrf]

All good @JavierCasares. Just wanted to make a note of it for the record as some of the above listed plugins are now misrepresented in a way.

And as I said before: I fully support the drive to get people/hosters to enable the Intl extension. Some of the UTF8 issues in WP have been bugging me for years and will be much easier to solve in a stable manner when using Intl. It would also be a great idea to look into moving from GetText to Intl for the translations.

My fingers are itching to use it, but I agree, first things first: get people to enable it.

[JavierCasares]

I just did a proposal for our Handbook.

​https://github.com/WordPress/hosting-handbook/pull/77

[JavierCasares]

We just publish this post talking about and recommending the installation for hosters.

​https://make.wordpress.org/hosting/2021/05/20/why-hosters-should-install-the-php-intl-extension/

It's open for discussion, and it's a go to recommend it for Site Health by the Hosting Team.

[josklever]

I've just added a request to make the Intl extension available in DirectAdmin CustomBuild for easy installation. It can be upvoted on:
​https://feedback.directadmin.com/b/feature-requests/php-intl-extension

EDIT: "It's bundled in by default in all the installations of PHP on DirectAdmin systems."

So nothing to do there. :-)

[zodiac1978]

Replying to JavierCasares:

We just publish this post talking about and recommending the installation for hosters.

​https://make.wordpress.org/hosting/2021/05/20/why-hosters-should-install-the-php-intl-extension/

It's open for discussion, and it's a go to recommend it for Site Health by the Hosting Team.

Would it make sense to update ​https://make.wordpress.org/hosting/handbook/server-environment/#php-extensions now to reflect the recommendation of the intl module? At the moment it still lists the intl module in the second section of modules which may help but not as a recommendation.

After clarification about the status of the package I remove the 2nd opinion tag.

@Clorith Are you fine with adding this as a recommendation in the next release? I could prepare the patch for this in this case.

[zodiac1978]

Add intl to the recommended but not required PHP extensions

[Clorith]

I think this is good to go, are there specific functions we should be keeping an eye out for and want to make sure are available that would benefit core in the future, or should we start by just checking for the extension in general?

With backwards compatibility in mind, we would need to add function availability checks before using any function that doesn't have a polyfill available any way (when/if we get to that point), so it may be enough to just look for the extension it self for now, as proposed in 52654.diff​, what do you think @jrf ?

[jrf]

Replying to Clorith:

I think this is good to go, are there specific functions we should be keeping an eye out for and want to make sure are available that would benefit core in the future, or should we start by just checking for the extension in general?

With backwards compatibility in mind, we would need to add function availability checks before using any function that doesn't have a polyfill available any way (when/if we get to that point), so it may be enough to just look for the extension it self for now, as proposed in 52654.diff​, what do you think @jrf ?

@Clorith I agree. For the time being, while Core is not using any of the Intl functions yet, nor using the polyfills, checking for the extension alone should be sufficient.

If, at some point in the (hopefully not too far away) future, the first uses of Intl functions will be introduced into Core, we can revisit if we should make this check more specific.

Any uses in Core, should, for the foreseeable future, either be accompanied by an appropriate polyfill; or be wrapped in functionality specific checks, i.e. function_exists()/class_exists(), rather than extension_loaded(), and have a fallback if the functionality is not available.

As a side-note: I just checked and the WP Core Docker container already ships with the intl extension.

Happy for this patch to go in as is. As it doesn't add any logic, I'm not concerned about there not being any tests (though adding more tests for the Site Health classes would be nice, but outside of the scope of this ticket).

[SergeyBiryukov]

In 51804:

Site Health: Add Intl to the list of recommended PHP extensions.

Why is the PHP Intl extension important?

WordPress is a global, international software, with support for a multitude of languages and with infinite combinations. Approximately half of the installations are in a language that is not the default (English), and this leads us to think about localization, transliteration, encoding conversions, calendar operations, collation… in short, everything you have with the different languages and formats that are around the planet. And this is what the ​PHP Intl extension provides.

What do we as the WordPress Community gain from this extension?

This extension provides a lot of functions for better internationalization support, including but not limited to:

• ​collator_compare() to compare Unicode text strings
• ​numfmt_format() to format a number according to the selected locale
• the ​normalization of characters
• the ​formatting of messages
• getting the ​first day of the week according to the locale.

In addition to functionality and ease of development, the extension can also help improve security, with classes like Spoofchecker that can tell you ​if ‘google.com’, ‘goog1e.com’ can confuse the user, or functions related to Internet domains, both to convert an ​IDN domain to text and ​text to IDN.

Hosting Team Recommendation

Taking into account that WordPress continues to grow, the Hosting Team has considered a good recommendation, but not an obligation, for all hosts that work with WordPress to offer this extension, by default, to all users.

Reference: ​Why hosters should install the PHP-intl extension.

Follow-up to [44986], [46268].

Props zodiac1978, JavierCasares, jrf, Clorith, josklever.
Fixes #52654.

[SergeyBiryukov]

In 51805:

Site Health: Move the Imagick entry higher in the list of recommended PHP extensions.

This better matches its position in the ​Hosting Team's handbook recommendations.

Follow-up to [44986], [46268], [51804].

See #52654.

[JavierCasares]

About the ImageMagick we should try to check if it's the version 6 or version 7. The newest can work with WEBP/WEBM and the new supported formats.

Moreover, a lot of OS have the version 6 by default because version 7 needs to be compiled manually (at least, at this moment).

[zodiac1978]

Replying to JavierCasares:

About the ImageMagick we should try to check if it's the version 6 or version 7. The newest can work with WEBP/WEBM and the new supported formats.

Moreover, a lot of OS have the version 6 by default because version 7 needs to be compiled manually (at least, at this moment).

As this ticket is about the intl extension and already fixed: Is there already a trac ticket for this new Imagick 6/7 version check?

I think this could be buried here, because the description is about something else.

[JavierCasares]

OK, yes, there is one now: #54116 ;)

[audrasjb]

In 53754:

Formatting: Normalize to Unicode NFC encoding before converting accent characters in remove_accents().

This changeset adds Unicode sequence normalization from NFD to NFC, via the normalizer_normalize() PHP function which is available with the recommended intl PHP extension.

This fixes an issue where NFD characters were not properly sanitized. It also provides a unit test for NFD sequences (alternate Unicode representations of the same characters).

Props NumidWasNotAvailable, targz, nacin, nunomorgadinho, p_enrique, gitlost, SergeyBiryukov, markoheijnen, mikeschroder, ocean90, pento, helen, rodrigosevero, zodiac1978, ironprogrammer, audrasjb, azaozz, laboiteare, nuryko, virgar, dxd5001, onnimonni, johnbillion.
Fixes #24661, #47763, #35951.
See #30130, #52654.

[jason_the_adams]

Speaking for GiveWP, I really hope this makes its way into core. We require the intl extension to properly format donation amounts into their appropriate number formatting based on the browser local. We also make use of the JavaScript Intl object for the same reason.

As we start Phase 3 of Gutenberg, we need to be getting ready for Phase 4 which starts to tackle internationalization and translation. If we get this into Site Health now, then it will give hosts more time to be ready, making it more reliable by the time Phase 4 begins.

[SergeyBiryukov]

Replying to jason_the_adams:

Speaking for GiveWP, I really hope this makes its way into core.

This was merged in [51804] for WordPress 5.9, great to see it being helpful :)

[jason_the_adams]

Replying to SergeyBiryukov:

This was merged in [51804] for WordPress 5.9, great to see it being helpful :)

So... what you're saying is that I should more carefully read things before replying to them? That seems like good advice. 😆

Glad to see it's in core! Thank you for being patient and taking the time to point that out to me!