Combining characters aren't normalized to Unicode Form C on output

[edent]

The HTML produced by WordPress functions like get_the_excerpt() and the_content() doesn't perform any Unicode Normalization.

This lack of normalisation is problematic because the W3C recommends Normalization Form C - ​https://www.w3.org/TR/charmod-norm/

It is possible to wrap these functions - for example normalizer_normalize( get_the_excerpt(), Normalizer::FORM_C ); - but this quickly becomes cumbersome in complex themes.

Would it make sense for WordPress to ensure that the content coming out of the post object is normalized before being returned?

Test

My blog post contains the text Anaïs. The ï is U+0069 LATIN SMALL LETTER I and U+0308 COMBINING DIAERESIS.

When I run the output of the_content() through the W3C Validator, it complains

Warning: Text run is not in Unicode Normalization Form C.

Background

There's this previous ticket which talks about normalizing in the editor and on save - #30130.

However, for anyone saving plain text / markdown this doesn't seem to work. Text is saved un-normalised.

Fixes #66093

Summary

• Normalize output from the_content() and get_the_excerpt() to Unicode NFC when the intl extension is available.
• Preserve output when normalization is unavailable or the input is not valid UTF-8.
• Add regression tests for both public output paths.

Testing

• vendor/bin/phpunit -c phpunit.xml.dist tests/phpunit/tests/post/getTheExcerpt.php
• vendor/bin/phpunit -c phpunit.xml.dist tests/phpunit/tests/post/getTheContent.php
• vendor/bin/phpcs --standard=phpcs.xml.dist on all four changed files
• git diff --check

The focused PHPUnit files passed: 18 tests, 26 assertions total.