HTML API: Fuzz-test and fix detections in 7.1

[dmsnell]

The purpose of this ticket is to track work to fuzz-test the HTML API against inputs that could raise issues or bugs and then fix them in 7.1.

Noted already are a few cases where:

• out-of-bounds text access throws warnings
• HTML normalization is non-idempotent on malformed inputs
• HTML normalization of prefixed attributes like xlink:href=… (incorrectly serialized as xlink href=…

Trac ticket: Core-65372

Fuzz-testing was performed against the HTML API for finding edge cases
that might be broken in the existing parsing code. A few issues were
discovered with HTML normalization and warnings from out-of-bounds
string reads.

This patch contains new tests catching regressions on these behaviors
and adds fixes for the discovered issues.

A special-case for FORM closing tags inside TABLEs was added after
investigation turned up that many normalization issues stem from this
single issue, where next_tag() would already fail to proceed, but the
normalization was continuing with already-created virtual tokens. This
special-case should be investigated as more support is added to the HTML
API to ensure that it couldn’t be removed for more robust core code.

[dmsnell]

In 62439:

HTML API: Fixes for issues discovered while fuzzing.

Fuzz-testing was performed against the HTML API for finding edge cases
that might be broken in the existing parsing code. A few issues were
discovered with HTML normalization and warnings from out-of-bounds
string reads.

This patch contains new tests catching regressions on these behaviors
and adds fixes for the discovered issues.

Patch proposed by Codex and revised by dmsnell.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/11982
Discussed in: https://core.trac.wordpress.org/ticket/65372

Fixes #65372.

[jonsurrell]

Reopening for additional fixes during 7.1 cycle. Additions to be included in description.

Fixes an issue where HTML Processor serialization issue where adjusted foreign attributes such as xlink:href were emitted with a space separator as xlink href. After normalization, this would become two attributes, xlink and href.

This keeps duplicate detection keyed to the adjusted qualified attribute name while converting adjusted foreign attributes back to colon-prefixed names when writing HTML. It also adds regression coverage for both WP_HTML_Processor::normalize() and serialize_token().

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5
Used for: Discover issue with fuzz testing, add the regression test, implement fix, perform verification, and draft this PR description.

### Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with ​Claude Code

### Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with ​Claude Code

### Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Verified that the space-to-colon replacement in ​`serialize_token()` is safe: the tokenizer never permits spaces in attribute names, so a space can only appear in a qualified name via the adjusted foreign attributes table in ​`get_qualified_attribute_name()`. Keeping duplicate detection keyed on the space-form qualified name is equivalent to keying on the serialized name, since attribute names from the tag processor are unique lowercase names and the mapping is collision-free.

🤖 Generated with ​Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

Trac ticket: Core-65372

Detected while fuzz-testing the UTF-8 handling code, this defect meant that the tests were verifying the wrong behavior. Namely, they verified a stringification of ASCII digits, which always converted plainly, when they were meant to test handling of invalid UTF-8 sequences.

This patch fixes the test by calling chr() on the byte values before concatenating into a big string.

Follow-up to: [60950].

Trac ticket: Core-65372

Found during fuzzing work on the HTML API and adjacent code. The previous version of this function used a Unicode PCRE to detect noncharacter code points, but that invocation failed if the input string contained sequences of invalid UTF-8 bytes.

This patch replaces the Unicode PCRE with a mapped sequence of raw bytes. This version works in environments without Unicode support and it works when invalid bytes are present, making it possible to remove the fallback function as well.

Follow-up to [61000].

[dmsnell]

In 62484:

Charset: Fix broken test for utf8_decode() fallback.

Detected while fuzz-testing the UTF-8 handling code, this defect meant that the tests were verifying the wrong behavior. Namely, they verified a stringification of ASCII digits, which always converted plainly, when they were meant to test handling of invalid UTF-8 sequences.

This patch fixes the test by calling chr() on the byte values before concatenating into a big string.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/12147
Discussed in: https://core.trac.wordpress.org/ticket/65372

Props dmsnell, jonsurrell.
Follow-up to: [60950].
See #65372.

Merged in [62484]
​https://github.com/wordpress/wordpress-develop/commit/6c363d4e227791de30632eb27b668a058106327a

[dmsnell]

In 62485:

Charset: Replace polyfill wp_has_noncharacters() with direct PCRE version.

Found during fuzzing work on the HTML API and adjacent code. The previous version of this function used a Unicode PCRE to detect noncharacter code points, but that invocation failed if the input string contained sequences of invalid UTF-8 bytes.

This patch replaces the Unicode PCRE with a mapped sequence of raw bytes. This version works in environments without Unicode support and it works when invalid bytes are present, making it possible to remove the fallback function as well.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/12148
Discussed in: https://core.trac.wordpress.org/ticket/65372

Follow-up to [61000].

Props dmsnell, jonsurrell.
See #65372.

Merged in [62485]
​https://github.com/wordpress/wordpress-develop/commit/eef6a6e9854febe3f5b4433992a2d0dd173bd962

Trac ticket: Core-65372

⚠️ Not sure if this is a positive fix or not because things get really messy when a site isn’t running UTF-8. For example, if the blog_charset is non-UTF-8, then the encoded code points won’t render on the frontend; they will be corrupted. However, if we encode in the configured charset, it’s likely that the code points aren’t represented in that charset, and will be rendered as ? instead, which is corrupt. It still makes most sense to run everything internally as UTF-8 and then only re-encode on transit to the browser if the site isn’t running UTF-8. In that case, this is a positive change.

This was brought up during fuzz testing of the HTML API. After polyfilling mb_chr() and relying on it in the HTML decoder, it became possible that for sites with a non-UTF-8 charset selected, then the creation of text from code points when decoding numeric character references might produce corrupted text, or text which encodes to non-UTF-8 bytes.

While for these sites, there are broader issues with non-UTF-8 support, this change ensures that code point encoding remains deterministic.

Follow-up to [62424].

[dmsnell]

In 62487:

HTML API: Ensure that code points always encode to UTF-8

This was brought up during fuzz testing of the HTML API. After
polyfilling mb_chr() and relying on it in the HTML decoder, it became
possible that for sites with a non-UTF-8 charset selected, then the
creation of text from code points when decoding numeric character
references might produce corrupted text, or text which encodes to
non-UTF-8 bytes.

While for these sites, there are broader issues with non-UTF-8 support,
this change ensures that code point encoding remains deterministic.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/12155
Discussed in: https://core.trac.wordpress.org/ticket/65372

Follow-up to [62424].

Props dmsnell, jonsurrell.
See #65372.

Merged in [62487]
​https://github.com/wordpress/wordpress-develop/commit/41c748932b65a01afcaa267e2d0609d7e355c7c1

[dmsnell]

In 62492:

HTML API: preserve adjusted foreign attributes on serialization.

Discovered during fuzz-testing of the HTML API. Adjusted foreign attributes, such as xlink:href, were being normalized with a space instead of a colon through ::serialize_token(). This led to the creation of two attributes on output instead of the proper singular attribute.

This patch corrects the issue by ensuring that the attribute namespace and name are separated by a colon when serializing.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/12140
Discussed in: https://core.trac.wordpress.org/ticket/65372

Props jonsurrell.
See #65372.

Merged in [62492]
​https://github.com/wordpress/wordpress-develop/commit/a513738f4860c2bfb5122234d754233c0c6246f7

</br> gets special treatment in the HTML standard. It creates a BR element with no attributes (regardless of whether it _appears_ to have attributes in the HTML like </br attr-name>).

• Stop on </br> in ::next_tag() like other openers.
• Return array() from ::get_attribute_names_with_prefix() on </br> instead of null like true close tags.
• Add tests.

To do:

• [ ] Ensure </br> works with functions like ::set_attribute() and ::add_class(). These will require rewriting the tag to <br attrs …> or the attributes will be discarded.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

Moved to trac ticket https://core.trac.wordpress.org/ticket/65372, this has been appearing in fuzz testing. I added a to-do, this likely needs some extra work to handle attribute modification. It will basically need to rewrite the entire token span I think because any attribute-like syntax needs to be removed before rewriting to <br> and only then can attributes be added.

I would like to move this forward when those issues are handled.

## Summary

• Reconsume a non-matching byte after < while scanning script data.
• Prevent less-than sequences before the real </script> closer from making normalization drop the script element.
• Add coverage for script less-than parsing and normalization cases.

​Inputs like `<script><</script>` or `<script>whatever… <</script>` would fail to parse at an incomplete token.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Testing

• Focused script parsing and normalization PHPUnit coverage passes.
• See HTML API debugger text example.

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5.5
Used for: PR description cleanup and code review.

Yes! HTML API fuzz tests caught this.

[jonsurrell]

In 62509:

HTML API: Ensure tag processor recognizes SCRIPT tag closers.

Address edge cases where SCRIPT tag closers were not detected and the processor remained paused on an incomplete token.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12184.

Props jonsurrell, dmsnell.
See #65372.

Merged in r62509.

XMP contents are not decoded and should not use HTML character reference escaping when serializing.

Escaping on serialize does not preserve the contents, so sequential normalizations of the original become:

1. <xmp> < > & \" ' ␀ </xmp> (original, ␀ is U+0000 NUL)
2. <xmp> &lt; &gt; &amp; \&quot; &apos; � </xmp>
3. <xmp> &amp;lt; &amp;gt; &amp;amp; \&amp;quot; &amp;apos; � </xmp>
4. etc…

(Note that the NUL -> � replacement is correct).

​See this demo

Note that XMP is an obsolete element, however it is still present in the HTML5 specification.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Testing

Compare the HTML API debugger example above on 7.0 with this branch.

## Use of AI Tools

AI assistance: Yes
Tool(s): Claude (Opus 4.8, Fable 5), Codex (GPT-5.5)
Used for: Fuzz test implementation, analysis of defects, fix implementation, review.

[jonsurrell]

In 62542:

HTML API: Ensure correct serialization of XMP contents.

The xmp element is parsed with the generic raw text element parsing algorithm, so its text content must be appended literally when serializing rather than escaped with HTML character references.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12193.

Props jonsurrell.
See #65372.

Merged in r62542.

Part of decoding HTML named character references in attribute values may involve ​checking the codepoint immediately following the named character reference:

13.2.5.73 Named character reference state
…

If there is a [named character reference] match

• If the character reference was consumed as part of an attribute, and the last character matched is not a U+003B SEMICOLON character (;), and the next input character is either a U+003D EQUALS SIGN character (=) or an ASCII alphanumeric, then, for historical reasons, flush code points consumed as a character reference and switch to the return state.

The ASCII alphanumeric check was implemented using ctype_alnum(). The behavior of this depends on the host system and the locale. On my system (macOS) it returns true for characters outside of the desired ASCII alphanumeric range.

php -r 'echo ctype_alnum( "\xC2" ) ? "Affected" : "Unaffected";'
// Affected

This change compares the following byte with the well-defined ASCII alphanumeric ranges from the HTML specification.

This change also does some minor restructuring of the method to make it align clearly with the specification and to include an early return and avoid the byte comparison in the majority of cases.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

I'm trying a revert of the ctype_alnum() change to see if there are any failures on CI.

On my system, I get these failures from one of the new tests:

1) Tests_HtmlApi_WpHtmlDecoder::test_semicolonless_legacy_reference_before_multibyte_attribute_follower with data set #0 ('&copy¯\_(ツ)_/¯', '©¯\_(ツ)_/¯', '©', 5)
Failed to decode the full attribute value as expected.
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'©¯\_(ツ)_/¯'
+'&copy¯\_(ツ)_/¯'

/Users/jonsurrell/a8c/wordpress-develop/trunk/tests/phpunit/tests/html-api/wpHtmlDecoder.php:121

2) Tests_HtmlApi_WpHtmlDecoder::test_semicolonless_legacy_reference_before_multibyte_attribute_follower with data set #1 ('&notಠ_ಠ', '¬ಠ_ಠ', '¬', 4)
Failed to decode the full attribute value as expected.
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'¬ಠ_ಠ'
+'&notಠ_ಠ'

/Users/jonsurrell/a8c/wordpress-develop/trunk/tests/phpunit/tests/html-api/wpHtmlDecoder.php:121

3) Tests_HtmlApi_WpHtmlDecoder::test_semicolonless_legacy_reference_before_multibyte_attribute_follower with data set #2 ('&nbsp£20', ' £20', ' ', 5)
Failed to decode the full attribute value as expected.
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-' £20'
+'&nbsp£20'

/Users/jonsurrell/a8c/wordpress-develop/trunk/tests/phpunit/tests/html-api/wpHtmlDecoder.php:121

4) Tests_HtmlApi_WpHtmlDecoder::test_semicolonless_legacy_reference_before_multibyte_attribute_follower with data set #3 ('&nbsp🎉', ' 🎉', ' ', 5)
Failed to decode the full attribute value as expected.
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-' 🎉'
+'&nbsp🎉'

/Users/jonsurrell/a8c/wordpress-develop/trunk/tests/phpunit/tests/html-api/wpHtmlDecoder.php:121

5) Tests_HtmlApi_WpHtmlDecoder::test_semicolonless_legacy_reference_before_multibyte_attribute_follower with data set #4 ('&reg™', '®™', '®', 4)
Failed to decode the full attribute value as expected.
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'®™'
+'&reg™'

/Users/jonsurrell/a8c/wordpress-develop/trunk/tests/phpunit/tests/html-api/wpHtmlDecoder.php:121

FAILURES!
Tests: 115, Assertions: 331, Failures: 5.

IFRAME, NOEMBED, and NOFRAMES contents were omitted from normalized HTML. Print their rawtext contents verbatim ​according to the standard for serializing HTML fragments.

If current node is a ​Text node
If the parent of current node is a ​style, ​script, ​xmp, ​iframe, ​noembed, ​noframes, or ​plaintext element, or if the parent of current node is a ​noscript element and ​scripting is enabled for the node, then append the value of current node's ​data literally.

Otherwise, append the value of current node's ​data, ​escaped as described below.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5.5
Used for: Detection and initial implementation.

<g attr=/> is not a self-closing tag. It is a tag with the [attr="/"] attribute.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

HTML PRE, LISTING, and TEXTAREA elements ignore a leading newline. Previous changes ensured those newline were preserved (r61747, r61754), but did not account for non-HTML examples of these elements (for example svg:textarea).

Trac ticket: https://core.trac.wordpress.org/ticket/65372
Follow-up to r61747, r61754

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex GPT-5.5
Used for: Fuzz testing, diagnosis, initial fix implementation, review and feedback.

Trac ticket: https://core.trac.wordpress.org/ticket/65372
Follow-up to r61754.

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex GPT-5.5
Used for: Fuzz testing, initial implementation, review and feedback.

Duplicate of ​https://github.com/WordPress/wordpress-develop/pull/10879

[jonsurrell]

In 62573:

HTML API: Replace locale-dependent ctype check in HTML decoder.

ctype_alnum() behaves differently depending on the host system and locale. Replace it with a direct ASCII byte comparison that behaves consistently across environments.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12286.

Props jonsurrell, dmsnell.
See #65372.

Committed in r62573.

[jonsurrell]

In 62574:

HTML API: Prevent HTML newline normalization on foreign elements.

HTML and foreign element normalization differ in some cases. Ensure the HTML-specific newline injection is not applied to foreign elements like svg:textarea.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12322.

Follow-up to [61747].

See #65372.

Committed in r62574.

Ensure foreign content elements do not match HTML elements on the stack of open element checks.

A closing HTML tag could close non-matching foreign content elements producing breadcrumgbs in some documents.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5.5
Used for: Fuzzing, diagnosis, fix implementation.

[jonsurrell]

In 62575:

HTML API: Respect namespace in open element lookup.

Prevent foreign elements from incorrectly satisfying checks for open HTML elements.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12353.

Props jonsurrell, dmsnell.
See #65372.

Merged in r62575.

There’s a case of removing an attribute and leaving an ambiguous trailing /> that came up in my fuzz-testing

Nice one, I'll see if variant of that is reproducible and add some unit tests.

I plan to land this and address that on its own. Attribute removal behavior with the self-closing flag are a different issue.

[jonsurrell]

In 62595:

HTML API: Ensure slash in attribute value is not a self-closing flag.

A trailing slash in an unquoted attribute value was incorrectly treated as a self-closing flag. For example, <div id=test/> is a tag with the id attribute value test/, not a self-closing tag.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12319.

Props jonsurrell, dmsnell.
See #65372.

Merged in r62595.

r62595 introduced an test code that asserts the wrong behavior. Fix it.

This fixed a test that is currently failing on trunk.

Trac ticket: https://core.trac.wordpress.org/ticket/65372
Follow-up to r62595.

## Use of AI Tools

[jonsurrell]

In 62598:

HTML API: Fix test assertion and add test case.

[62595] introduced a test assertion for incorrect behavior. Correct the assertion and add another case.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12367.

Props dmsnell.
See #65372. Follow-up to [62595].

Merged in r62598.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

There’s a case of removing an attribute and leaving an ambiguous trailing /> that came up in my fuzz-testing, but I may have fixed it already.

<div a b=3 /b>

<div a />

something like this

Work-in-progress: ​https://github.com/WordPress/wordpress-develop/pull/12368

Prevent modifying the by-reference variable when WP_HTML_Decoder::read_character_reference() misses a named reference.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5.5
Used for: Fuzz tests, diagnosis, implementation.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

The tag processor defers some processing. It is currently applied inconsistently.

Newline should be normalized (CRLF and CR become LF). This is from the input stream pre-processing phase.

Null bytes are replaced with U+FFFD replacement. This depends on the tokenization rules, but relevant for this PR is that it applies in tag names, attribute names, and attribute values.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5.5
Used for: Fuzz testing, diagnosis, initial implementation, review.

Failing tests are currently failing on trunk:

There was 1 failure:

1) WP_Test_REST_Attachments_Controller::test_sideload_animated_video_companions_write_metadata
Sideloading animated_video should succeed.
Failed asserting that 400 is identical to 200.

[jonsurrell]

In 62667:

HTML API: Apply deferred byte processing at read interfaces.

Ensure that input processing (intentionally deferred by the HTML API) is applied consistently. This includes newline normalization (CRLF and CR become LF) and null byte replacement (U+FFFD in tag names, attribute names, and attribute values).

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12385.

Props jonsurrell, dmsnell.
See #65372.

Merged in r62667.

## Summary

• Serialize decoded carriage returns in text, attributes, and RCDATA-style contents as 
 so normalization reaches a fixed point.
• Built on r62667.

## Testing

• Regression coverage for decoded CR handling across text, attributes, RCDATA, tables, and templates.
• HTML API and html5lib PHPUnit groups and PHPCS pass.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5.5
Used for: PR description cleanup and code review.

[jonsurrell]

In 62711:

HTML API: Ensure attribute value prefix matches the full search text.

WP_HTML_Decoder::attribute_starts_with() incorrectly reported a match when the attribute value was exhausted before the search text.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12380.

Props jonsurrell, dmsnell.
See #65372.

Merged in r62711.

[jonsurrell]

In 62715:

HTML API: Preserve raw text contents in serialization.

The contents of iframe, noembed, and noframes elements are parsed as raw text and must be appended literally when serializing rather than escaped or omitted. Setting the modifiable text of these elements, as well as xmp, is now supported.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12289.

Follow-up to [62542].

Props jonsurrell, dmsnell.
See #65372.

Merged in r62715.

[jonsurrell]

In 62716:

HTML API: Ensure carriage returns are serialized in HTML.

HTML parsers always replace carriage returns with line feeds when normalizing input. This ensures that serializing or normalizing an HTML document with the HTML API will preserve escaped U+000D CARRIAGE RETURN in the output.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/12466.

See #65372.

Merged in r62716.

Removing an attribute could accidentally change tag semantics. For example, removing attr from <svg><g /attr>ok produced <svg><g />ok, turning the g element into a self-closing tag.

The / is outside the recorded attribute span, so deleting the span exposed it directly before >. This keeps a separating space when removing a slash-adjacent attribute. Duplicate and repeated removals are covered and remain idempotent.

Tests were added first for primary, duplicate, and repeated removals. The Tag Processor and full HTML API test suites pass.

Trac ticket: https://core.trac.wordpress.org/ticket/65372

Alternative to #12368.

## Use of AI Tools

AI assistance: Yes
Tool(s): Codex
Model(s): GPT-5
Used for: Diagnosis, tests, implementation, adversarial review, and verification. All generated changes were reviewed before submission.

Closing in favor of ​https://github.com/WordPress/wordpress-develop/pull/12518.