WordPress.org

Make WordPress Core

Opened 10 months ago

Closed 3 months ago

Last modified 3 months ago

#52654 closed enhancement (fixed)

Check and recommend intl extension in Site Health

Reported by: zodiac1978 Owned by: SergeyBiryukov
Milestone: 5.9 Priority: normal
Severity: normal Version:
Component: Site Health Keywords: has-patch commit
Focuses: Cc:

Description

First reported here: https://github.com/WordPress/health-check/issues/333

I suggest to check for the existence of the intl PHP extension with the status of a recommendation.

This extension is necessary to use the Normalizer function which would solve many problems with internationalization issues for special characters:
https://www.php.net/manual/en/intl.requirements.php

Unfortunately this extension is not included per default on every PHP/hoster.

Related issues: #30130, #35951, #24661, #47763

Additional info from the Github issue:

intl has hard dependency on icu system libraries, which will consume over 30 Mb diskspace on smaller (VPS/embedded) systems. Each loaded library claims some memory and initialization time overhead.

Bigger distributions can link libicu to libxml2 library and therefore it can be loaded by libxml2 linked extensions, but this dependency is optional

The site health check for this extension could be easily added like shown here:
https://gist.github.com/Zodiac1978/ce9db99becd3507ec8d8207c583e947b

This was discussed in the hosting team with some additional comments:
https://make.wordpress.org/hosting/?s=%22intl+extension%22

Especially this feedback from @swissspidy is interesting:

in the case of the INTL extension, when a server is running PHP 7.2 but uses a super outdated version of the extension which doesn’t include the functions and constants one would expect. So any site health check for extensions should also check against these things.

Therefore there are two ways to check here. We could check for the extension itself or we could check for the function(s) we need in core, like the Normalize function which could help resolve the bug tickets mentioned above.

I would recommend the latter.

Any input on how to get forward here from the Hosting team and the Site Health people is much appreciated!

Attachments (1)

52654.diff (447 bytes) - added by zodiac1978 4 months ago.
Add intl to the recommended but not required PHP extensions

Download all attachments as: .zip

Change History (37)

#1 follow-up: @Clorith
8 months ago

  • Keywords 2nd-opinion added
  • Type changed from defect (bug) to enhancement

Is the Normalizer class still maintained and going to be a thing moving forward?

A quick look at the PHP documentation shows it's available in PHP 5.3 and PHP 7, but not PHP 8, but then rather as an optional install via the intl PECL package.

The latter makes it seem like a non-starter based on the linked Hosting Team discussion on criteria for recommendations, but I'm open to it if the Hosting Team feels strongly about this as well, as the ones most likely to get requests for this.

This ticket was mentioned in Slack in #hosting-community by clorith. View the logs.


8 months ago

#3 follow-up: @JavierCasares
8 months ago

In the "intl" there are several functions.... some summary of use:

+ Locale

  • WooCommerce
  • Yoast SEO
  • Jetpack
  • Contact Form 7
  • Wordfence Security
  • ...

+ Normalizer

  • Contact Form by WPForms
  • Facebook for WooCommerce
  • BackWPup
  • File Manager
  • Beaver Builder
  • ...

+ MessageFormatter

  • UpdraftPlus
  • Beaver Builder
  • Toolset Types
  • Post SMTP Mailer
  • Mollie Payments for WooCommerce
  • ...

+ Collator

  • WooCommerce

+ NumberFormatter

  • WP RSS Aggregator
  • Matomo Analytics
  • Unite Gallery Lite
  • Amelia

+ IntlTimeZone

  • MailPoet
  • WP RSS Aggregator

+ IntlDateFormatter

  • MailPoet
  • WP RSS Aggregator
  • CAOS

+ Spoofchecker

  • MailPoet

+ Transliterator

  • GiveWP

+ UConverter

  • WPeMatico RSS Feed Fetcher
  • CyberSEO Lite
  • WordPress RSS Feed Retriever

+ IntlChar

  • Simply Static
  • Smart Grid-Layout Design for Contact Form 7

#4 in reply to: ↑ 1 @zodiac1978
8 months ago

Replying to Clorith:

Is the Normalizer class still maintained and going to be a thing moving forward?

A quick look at the PHP documentation shows it's available in PHP 5.3 and PHP 7, but not PHP 8, but then rather as an optional install via the intl PECL package.

The latter makes it seem like a non-starter based on the linked Hosting Team discussion on criteria for recommendations, but I'm open to it if the Hosting Team feels strongly about this as well, as the ones most likely to get requests for this.

As I understand it, it was developed in the past as PECL package:
https://pecl.php.net/package/intl

But nowadays it is an official part of PHP:
https://www.php.net/manual/en/intro.intl.php

The underlying code was in both cases the maintained ICU library:
http://site.icu-project.org/home

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


8 months ago

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


8 months ago

This ticket was mentioned in Slack in #hosting-community by jadonn. View the logs.


8 months ago

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


7 months ago

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


7 months ago

This ticket was mentioned in Slack in #hosting-community by crixu. View the logs.


7 months ago

This ticket was mentioned in Slack in #hosting-community by amykamala. View the logs.


7 months ago

#12 @JavierCasares
7 months ago

After the last session and after several meetings, the Hosting team has considered that the installation of the PHP-intl extension should be recommended (but not mandatory).

https://make.wordpress.org/hosting/2021/05/06/hosting-team-meeting-summary-2021-05-05/

#13 @jrf
7 months ago

A quick look at the PHP documentation shows it's available in PHP 5.3 and PHP 7, but not PHP 8, but then rather as an optional install via the intl PECL package.

@Clorith You may have looked before the manual was updated, but yes, the Intl extension is available in PHP 8 and it also states as much in the manual (at the time of writing):

(PHP 5 >= 5.3.0, PHP 7, PHP 8, PECL intl >= 1.0.0)

More relevant is that PHP needs to be build with `--enable-intl` for it to be available, so a functionality check (function_exists/class_exists/defined etc) is always needed before any of the Intl functionality can be used..

#14 @jrf
7 months ago

Oh and just FYI, the non-availability could - for a limited set of functionality - be worked around by using the Symfony polyfills for the Intl extension.

Last edited 7 months ago by jrf (previous) (diff)

#15 @zodiac1978
7 months ago

@jrf Thank you for the details about the extension and why it is not always available!

The polyfill could be useful for all the mentioned tickets, I think:

Related issues: #30130, #35951, #24661, #47763

Otherwise there is a chicken/egg-probleme here. Core does not use it, because it is not always available. Hoster do not implement it, because it is not required/used in core.

Hopefully this recommendation from the hosting team and the polyfill could be a starting point for these tickets. This would be a huge win for all non English speaking people using WordPress.

#16 in reply to: ↑ 3 ; follow-up: @jrf
7 months ago

Replying to JavierCasares:

In the "intl" there are several functions.... some summary of use:

I've just had a quick look through the search results and feel like I need to add a word of warning on the value of these as there appear to be quite some false positives due to the searches not being precise enough.

That by no means invalidates the drive to get people to activate the extension as I totally agree the Intl extension really does add significant value if we could use it.

It just means that the "plugin use" information as part of the making the case for recommending Intl is flawed.

+ Locale

Checked the complete first page and only found 8 legitimate uses out of 100 entries. Everything else was false positive, including the whole top five listed above. Most JS or PO files.

+ Normalizer

Verified first x entries and found mostly correct. About 25% false positives from the Guzzle README.md file, some more false positives for namespaced classes also called Normalizer`. Rest mostly coming from the Symfony Iconv/Intl polyfills.

+ MessageFormatter

Verified first x entries and nearly all are false positives. Mostly use in comments or the Guzzle UPGRADING.md file. Only found two legitimate uses from the Symfony Intl polyfill.

+ Collator

Verified first x entries and found mostly correct. About half coming from Symfony Intl Polyfill, the other half from a "free world map" function. Saw at least one false positive in a JS file.

+ NumberFormatter

Verified first x entries and found mostly correct. Saw at least one false positive in a JS file.

+ IntlTimeZone

Verified first x entries and found correct. About half coming from Twig, the other half from Symfony Validator.

+ IntlDateFormatter

Verified first x entries and found correct. About half coming from Twig.

+ Spoofchecker

Verified and found correct. Most coming from the Egulias\EmailValidator package.

+ Transliterator

Some false positives for use in comments or translatable strings. Some real use.

+ UConverter

Verified and found correct. Mostly found in a SimplePie code (not the WP Core version).

+ IntlChar

Verified and found correct.

#17 in reply to: ↑ 16 @JavierCasares
7 months ago

Replying to jrf:

Yes, we checked that before doing the Hosting Meeting. Also, we saw that the PHPMailer inside the core has some "intl" functions (with an alternative IF in all the cases).

The thing is Core / Plugins doesn't want to include some "intl" functions because a lot of hosters doesn't have the extension, so we are recommending to start installing it, so it will be available to use.

We will add it to the Hosting Handbook soon.

Sorry for the first confusion, but we checked it later in our meetings.

#18 @jrf
7 months ago

All good @JavierCasares. Just wanted to make a note of it for the record as some of the above listed plugins are now misrepresented in a way.

And as I said before: I fully support the drive to get people/hosters to enable the Intl extension. Some of the UTF8 issues in WP have been bugging me for years and will be much easier to solve in a stable manner when using Intl. It would also be a great idea to look into moving from GetText to Intl for the translations.

My fingers are itching to use it, but I agree, first things first: get people to enable it.

This ticket was mentioned in Slack in #hosting-community by jadonn. View the logs.


7 months ago

#21 follow-up: @JavierCasares
7 months ago

We just publish this post talking about and recommending the installation for hosters.

https://make.wordpress.org/hosting/2021/05/20/why-hosters-should-install-the-php-intl-extension/

It's open for discussion, and it's a go to recommend it for Site Health by the Hosting Team.

#22 @josklever
7 months ago

I've just added a request to make the Intl extension available in DirectAdmin CustomBuild for easy installation. It can be upvoted on:
https://feedback.directadmin.com/b/feature-requests/php-intl-extension

EDIT: "It's bundled in by default in all the installations of PHP on DirectAdmin systems."

So nothing to do there. :-)

Last edited 7 months ago by josklever (previous) (diff)

#23 in reply to: ↑ 21 @zodiac1978
4 months ago

  • Keywords needs-patch added; 2nd-opinion removed

Replying to JavierCasares:

We just publish this post talking about and recommending the installation for hosters.

https://make.wordpress.org/hosting/2021/05/20/why-hosters-should-install-the-php-intl-extension/

It's open for discussion, and it's a go to recommend it for Site Health by the Hosting Team.

Would it make sense to update https://make.wordpress.org/hosting/handbook/server-environment/#php-extensions now to reflect the recommendation of the intl module? At the moment it still lists the intl module in the second section of modules which may help but not as a recommendation.

After clarification about the status of the package I remove the 2nd opinion tag.

@Clorith Are you fine with adding this as a recommendation in the next release? I could prepare the patch for this in this case.

@zodiac1978
4 months ago

Add intl to the recommended but not required PHP extensions

#24 @zodiac1978
4 months ago

  • Keywords has-patch added; needs-patch removed

#25 @SergeyBiryukov
4 months ago

  • Milestone changed from Awaiting Review to 5.9

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


4 months ago

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


3 months ago

This ticket was mentioned in Slack in #hosting-community by jadonn. View the logs.


3 months ago

#29 follow-up: @Clorith
3 months ago

I think this is good to go, are there specific functions we should be keeping an eye out for and want to make sure are available that would benefit core in the future, or should we start by just checking for the extension in general?

With backwards compatibility in mind, we would need to add function availability checks before using any function that doesn't have a polyfill available any way (when/if we get to that point), so it may be enough to just look for the extension it self for now, as proposed in 52654.diff, what do you think @jrf ?

#30 in reply to: ↑ 29 @jrf
3 months ago

  • Keywords commit added

Replying to Clorith:

I think this is good to go, are there specific functions we should be keeping an eye out for and want to make sure are available that would benefit core in the future, or should we start by just checking for the extension in general?

With backwards compatibility in mind, we would need to add function availability checks before using any function that doesn't have a polyfill available any way (when/if we get to that point), so it may be enough to just look for the extension it self for now, as proposed in 52654.diff, what do you think @jrf ?

@Clorith I agree. For the time being, while Core is not using any of the Intl functions yet, nor using the polyfills, checking for the extension alone should be sufficient.

If, at some point in the (hopefully not too far away) future, the first uses of Intl functions will be introduced into Core, we can revisit if we should make this check more specific.

Any uses in Core, should, for the foreseeable future, either be accompanied by an appropriate polyfill; or be wrapped in functionality specific checks, i.e. function_exists()/class_exists(), rather than extension_loaded(), and have a fallback if the functionality is not available.

As a side-note: I just checked and the WP Core Docker container already ships with the intl extension.

Happy for this patch to go in as is. As it doesn't add any logic, I'm not concerned about there not being any tests (though adding more tests for the Site Health classes would be nice, but outside of the scope of this ticket).

#31 @SergeyBiryukov
3 months ago

  • Owner set to SergeyBiryukov
  • Resolution set to fixed
  • Status changed from new to closed

In 51804:

Site Health: Add Intl to the list of recommended PHP extensions.

Why is the PHP Intl extension important?

WordPress is a global, international software, with support for a multitude of languages and with infinite combinations. Approximately half of the installations are in a language that is not the default (English), and this leads us to think about localization, transliteration, encoding conversions, calendar operations, collation… in short, everything you have with the different languages and formats that are around the planet. And this is what the PHP Intl extension provides.

What do we as the WordPress Community gain from this extension?

This extension provides a lot of functions for better internationalization support, including but not limited to:

In addition to functionality and ease of development, the extension can also help improve security, with classes like Spoofchecker that can tell you if ‘google.com’, ‘goog1e.com’ can confuse the user, or functions related to Internet domains, both to convert an IDN domain to text and text to IDN.

Hosting Team Recommendation

Taking into account that WordPress continues to grow, the Hosting Team has considered a good recommendation, but not an obligation, for all hosts that work with WordPress to offer this extension, by default, to all users.

Reference: Why hosters should install the PHP-intl extension.

Follow-up to [44986], [46268].

Props zodiac1978, JavierCasares, jrf, Clorith, josklever.
Fixes #52654.

#32 @SergeyBiryukov
3 months ago

In 51805:

Site Health: Move the Imagick entry higher in the list of recommended PHP extensions.

This better matches its position in the Hosting Team's handbook recommendations.

Follow-up to [44986], [46268], [51804].

See #52654.

#33 follow-up: @JavierCasares
3 months ago

About the ImageMagick we should try to check if it's the version 6 or version 7. The newest can work with WEBP/WEBM and the new supported formats.

Moreover, a lot of OS have the version 6 by default because version 7 needs to be compiled manually (at least, at this moment).

#34 in reply to: ↑ 33 @zodiac1978
3 months ago

Replying to JavierCasares:

About the ImageMagick we should try to check if it's the version 6 or version 7. The newest can work with WEBP/WEBM and the new supported formats.

Moreover, a lot of OS have the version 6 by default because version 7 needs to be compiled manually (at least, at this moment).

As this ticket is about the intl extension and already fixed: Is there already a trac ticket for this new Imagick 6/7 version check?

I think this could be buried here, because the description is about something else.

#35 @JavierCasares
3 months ago

OK, yes, there is one now: #54116 ;)

This ticket was mentioned in Slack in #hosting-community by javier. View the logs.


3 months ago

Note: See TracTickets for help on using tickets.