Changeset 60793 for trunk/src/wp-includes/compat-utf8.php

Timestamp:

09/23/2025 03:34:20 AM (6 months ago)

Author:

dmsnell

Message:

Charset: Improve UTF-8 scrubbing ability via new UTF-8 scanning pipeline.

This is the fourth in a series of patches to modernize and standardize UTF-8 handling.

wp_check_invalid_utf8() has long been dependent on the runtime configuration of the system running it. This has led to hard-to-diagnose issues with text containing invalid UTF-8. The function has also had an apparent defect since its inception: when requesting to strip invalid bytes it returns an empty string.

This patch updates the function to remove all dependency on the system running it. It defers to the mbstring extension if that’s available, falling back to the new UTF-8 scanning pipeline.

To support this work, wp_scrub_utf8() is created with a proper fallback so that the remaining logic inside of wp_check_invalid_utf8() can be minimized. The defect in this function has been fixed, but instead of stripping the invalid bytes it will replace them with the Unicode replacement character for stronger security guarantees.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/9498
Discussed in https://core.trac.wordpress.org/ticket/63837

Follow-up to: [60768].
Props askapache, chriscct7, Cyrille37, desrosj, dmsnell, helen, jonsurrell, kitchin, miqrogroove, pbearne, shailu25.
Fixes #63837, #29717.
See #63863.

File:

• trunk/src/wp-includes/compat-utf8.php (modified) (2 diffs)

trunk/src/wp-includes/compat-utf8.php

@@ -228 +228 @@
  * Fallback mechanism for safely validating UTF-8 bytes.
  *
- * @see wp_is_valid_utf8()
- *
  * @since 6.9.0
  * @access private
+ *
+ * @see wp_is_valid_utf6()
  *
  * @param string $bytes String which might contain text encoded as UTF-8.
@@ -249 +249 @@
     return $bytes_length === $next_byte_at && 0 === $invalid_length;
 }
+
+/**
+ * Fallback mechanism for replacing invalid spans of UTF-8 bytes.
+ *
+ * Example:
+ *
+ *     'Pi�a' === _wp_scrub_utf8_fallback( "Pi\xF1a" ); // “ñ” is 0xF1 in Windows-1252.
+ *
+ * @since 6.9.0
+ * @access private
+ *
+ * @see wp_scrub_utf8()
+ *
+ * @param string $bytes UTF-8 encoded string which might contain spans of invalid bytes.
+ * @return string Input string with spans of invalid bytes swapped with the replacement character.
+ */
+function _wp_scrub_utf8_fallback( string $bytes ): string {
+    $bytes_length   = strlen( $bytes );
+    $next_byte_at   = 0;
+    $was_at         = 0;
+    $invalid_length = 0;
+    $scrubbed       = '';
+
+    while ( $next_byte_at <= $bytes_length ) {
+        _wp_scan_utf8( $bytes, $next_byte_at, $invalid_length );
+
+        if ( $next_byte_at >= $bytes_length ) {
+            if ( 0 === $was_at ) {
+                return $bytes;
+            }
+
+            return $scrubbed . substr( $bytes, $was_at, $next_byte_at - $was_at - $invalid_length );
+        }
+
+        $scrubbed .= substr( $bytes, $was_at, $next_byte_at - $was_at );
+        $scrubbed .= "\u{FFFD}";
+
+        $next_byte_at += $invalid_length;
+        $was_at        = $next_byte_at;
+    }
+
+    return $scrubbed;
+}