Charset: Polyfill mb_ord() and mb_chr() for UTF-8

[dmsnell]

These functions are useful primitives but missing when the mbstring extension isn’t available. This patch adds polyfills for those few environments where this is the case so that WordPress code can unconditionally call them.

Only UTF-8 is supported for practical reasons. Code in unsupported environments should convert first to UTF-8 and then call these functions. See also #62172

This work is part of #31992

Trac ticket: Core-65342

See also Core-62172
Preparation for #11567

These functions are useful primitives but missing when the mbstring extension isn’t available. This patch adds polyfills for those few environments where this is the case so that WordPress code can unconditionally call them.

[dmsnell]

In 62424:

Charset: Polyfill mb_ord() and mb_chr().

These functions are useful primitives but missing when the mbstring
extension isn’t available. This patch adds polyfills for those few
environments where this is the case so that WordPress code can
unconditionally call them.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/11965
Discussed in: https://core.trac.wordpress.org/ticket/65342

Fixes #65342.

[dmsnell]

In 62425:

Charset: Update antispambot to handle multibyte characters.

In preparation for handling Unicode email addresses (non-US-ASCII
characters in the mailbox name), the antispambot() function needs to
be multi-byte aware so that it creates proper HTML numeric character
references and percent-encoded strings.

Previously it has been scanning the input email address byte-by-byte,
but with multibyte characters this will produce invalid sequences of the
transformations by encoding individual bytes of a multi-byte sequence as
if they were whole characters on their own.

This patch relies on the newly-polyfilled mb_ord() function and the
_wp_scan_utf8() function to crawl through an input email by code
point, assuming UTF-8 encoding. This ensures proper transformation.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/11567
Discussed in: https://core.trac.wordpress.org/ticket/31992

Props agulbra, akirk, benniledl, dmsnell, siliconforks.
See #65342.

This is a follow-up to ​https://github.com/WordPress/wordpress-develop/pull/11965.

There is a PHPStan rule level 0 error introduced in that PR:

 ------ ----------------------------------------------------------------------------- 
  Line   src/wp-includes/compat.php                                                   
 ------ ----------------------------------------------------------------------------- 
  227    Function _mb_ord() should return int|false but return statement is missing.  
         🪪  return.missing                                                           
         at src/wp-includes/compat.php:227                                            
 ------ ----------------------------------------------------------------------------- 

While it doesn't seem that $byte_length can ever be anything other than int<1, 4> when 1 !== $found_count, this is not picked up by PHPStan, leading ot the error.

An alternative to this would be to declare the type:

src/wp-includes/compat.php

@@ -221 +221 @@
                 return false;
         }
 
+        /** @var int<1, 4> $byte_length */
+
         // These are valid code points, so no further validation is required.
         $b0 = ord( $string[0] );

But this seems less clean.

Trac ticket: https://core.trac.wordpress.org/ticket/65342

## Use of AI Tools

None

[westonruter]

Follow-up with PHPStan fix: ​https://github.com/WordPress/wordpress-develop/pull/12020

Thanks @westonruter — the inline comment is probably better semantically but the return false would potentially make up for any regressions that would creep into _wp_scan_utf8(), as unlikely as those are.

[westonruter]

In 62436:

Charset: Add missing return statement to _mb_ord().

This fixes a return.missing PHPStan error in _mb_ord(), fixing the only rule level 0 violation currently reported. In practice the return is in an unreachable code path, but static analysis may not be aware of this.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12020.
Follow-up to r62424.

Props westonruter, dmsnell.
See #65342.

Committed in r62436 (​https://github.com/WordPress/wordpress-develop/commit/137eb30fc52bc59a5f55cf47bd75189b1e089a5a)

[tyxla]

@dmsnell @westonruter: native mb_chr() and mb_ord() accept alternative UTF-8 names such as utf-8 and utf8. Polyfills require exact UTF-8, returning false for these valid names. Should both checks use _is_utf8_charset()? We may also want to add tests for those.

[dmsnell]

@tyxla it’s probably fine if you want to do that. my thinking at the time was about normalizing UTF-8 and removing the confusion from supporting different labels. I believe that all existing uses in Core are either called without the charset parameter or the string literal UTF-8.

Did you encounter this as a bug anywhere?

[tyxla]

Normalizing makes sense to me, FWIW.

Did you encounter this as a bug anywhere?

No, but I've seen those variations of the labels used liberally before, and thought it might be cheap to support that in the polyfills.