PHP 8.1.: the default value of the flags parameter for htmlentities() et all needs to be explicitly set

[jrf]

From the PHP 8.1 changelog:

htmlspecialchars(), htmlentities(), htmlspecialchars_decode(),
html_entity_decode() and get_html_translation_table() now use
ENT_QUOTES | ENT_SUBSTITUTE rather than ENT_COMPAT by default. This means
that ' is escaped to ' while previously it was left alone.
Additionally, malformed UTF-8 will be replaced by a Unicode substitution
character, instead of resulting in an empty string.

Ref: ​https://github.com/php/php-src/blob/28a1a6be0873a109cb02ba32784bf046b87a02e4/UPGRADING#L149-L154

If effect this means that the output of the above mentioned functions may be different depending on the PHP version and the passed text string, unless the $flags parameter is explicitly passed.

I've run an initial scan over WordPress core with a new (not yet published) sniff for PHPCompatibility and this flags 33 issues.

• 1 issue in GetID3 which should be fixed upstream and the copy of GetID3 used in WP should be updated once the issue is fixed.
• 1 issue in PHPMailer which should be fixed upstream and the copy of PHPMailer used in WP should be updated once the issue is fixed.
• 1 issue in SimplePie which should be fixed upstream and the copy of SimplePie used in WP should be updated once the issue is fixed.
• And 30 issues in WP Core native code or code from external dependencies which are no longer maintained externally.

Detailed issue list: ​https://gist.github.com/jrfnl/9d56b4053faa62a0fe91dea1b14839bf

To fix this issue, the $flags parameter should be explicitly passed in each of these function calls.

Some investigation will be needed for each of these instances to determine what will be the optimal value for $flags.

Take note that the "old" parameter default in the function signature is documented as ENT_COMPAT, while in the parameter detail documentation, it states that the default, in actual fact, is ENT_COMPAT | ENT_HTML401.

However, by the looks of it, the full range of flag constants is available to us, which is at least one less problem.
There is no mention of any of the flags being added since PHP 5.6.
Ref: ​https://php-legacy-docs.zend.com/manual/php5/en/string.constants

It is strongly recommended to make sure that for each of these at least one unit test exists which exposes the difference in output between PHP < 8.1 and PHP 8.1 to safeguard the fixes which will be added for the future.

Also see:

• ​https://www.php.net/manual/en/function.htmlentities.php
• ​https://www.php.net/manual/en/function.html-entity-decode.php
• ​https://www.php.net/manual/en/function.htmlspecialchars.php
• ​https://www.php.net/manual/en/function.htmlspecialchars-decode.php
• ​https://www.php.net/manual/en/function.get-html-translation-table.php

[jrf]

PR for PHPMailer: ​https://github.com/PHPMailer/PHPMailer/pull/2365

[hellofromTonya]

With 4 days left until 5.9 Beta 1, moving this to 6.0.

[GaryJ]

According to ​https://www.php.net/manual/en/function.htmlentities.php, the default for flags for PHP 8.1 was "changed from ENT_COMPAT to ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML401".

I don't know what difference that ENT_HTML401 might make (there's also a ENT_HTML5 flag available as well), but might be something else to consider.

[jrf]

Replying to GaryJ:

According to ​https://www.php.net/manual/en/function.htmlentities.php, the default for flags for PHP 8.1 was "changed from ENT_COMPAT to ENT_QUOTES | ENT_SUBSTITUTE | ENT_HTML401".

I don't know what difference that ENT_HTML401 might make (there's also a ENT_HTML5 flag available as well), but might be something else to consider.

The ENT_HTML401 makes no difference as the value of that constant is 0, so the "bit" value remains the same with or without.
The ENT_HTML401 part is not an actual change, just better documentation.

See:

• ​https://github.com/php/doc-en/pull/361#issuecomment-925931381
• ​https://github.com/php/php-src/pull/7514

[costdev]

With 6.0 RC1 tomorrow, I'm moving this to the 6.1 milestone.

[rafiahmedd]

I think we can discuss this ticket on the next dev chat on slack. And it seems to be an important one.

[jrf]

FYI: If I remember correctly, @SergeyBiryukov is working on patches for this ticket together with other people from the Yoast WP Core team.

[SergeyBiryukov]

Yes, @aristath, @justinahinon, @poena, @afercia, and I worked on this recently in group sessions and should have a PR ready for review soon :)

[audrasjb]

With WP 6.1 RC 1 scheduled tomorrow (Oct 10, 2022), there is not much time left to address this ticket. Given it still needs a patch, let's move this ticket to 6.2.

Ps: if you were about to send a patch and if you feel it is realistic to commit it in the next few hours, please feel free to move this ticket back to milestone 6.1.

Trac ticket: https://core.trac.wordpress.org/ticket/53465

[costdev]

This ticket was discussed during the recent bug scrub.

@SergeyBiryukov Do you have time to give the PR a quick bump to see results on GitHub Actions?

[hellofromTonya]

The last scheduled 6.2 beta happened today. RC1 is next week. Moving this ticket to 6.3 and self-assigning ownership to help shepherd its completion.

[chaion07]

Thanks @jrf for continuing to work on this. We reviewed this ticket during a recent bug-scrub session and based on the feedback we are updating the milestone to 6.4. Cheers!

Props to @costdev & @mukesh27

[hellofromTonya]

Marking this work as a PHP 8.1 incompatibility. This is done for tracking purposes and for identifying documented exceptions to Core being compatible with PHP 8.1.

These functions will be included in the list of PHP 8.1 "compatible with documented exceptions" for WP 6.3.0 and beyond until resolved. See ​https://make.wordpress.org/core/2023/06/20/proposal-criteria-for-removing-beta-support-from-each-php-8-version/.