HTML API: Text nodes may be incorrectly split

[jonsurrell]

Certain inputs cause WP_HTML_Tag_Processor::next_token() to split text nodes that should be a single text node.

Example input: test< /A>

This should produce a single text node of test< /A> (via ::get_modifiable_text()). Instead it finds two adjacent text nodes of test immediately followed by < /A>.

​See the single text node parsed here.

This was found thanks to the external test suite proposed in #60227.

[dmsnell]

In 57489:

HTML API: Fix splitting single text node.

When next_token() was introduced, it brought a subtle bug. When encountering a < in the HTML stream which did not lead to a tag or comment or other token, it was treating the full text span to that point as one text node, and the following span another text node.

The entire span should be one text node.

In this patch the Tag Processor properly detects this scenario and combines the spans into one text node.

Follow-up to [57348]

Props jonsurrell
Fixes #60385

Fix an issue where a < character will always break a text node.

It should not break a text node if we will not start another node type, i.e. we find < followed by !, /, ? or an ascii alpha character.

Trac ticket: https://core.trac.wordpress.org/ticket/60385

[jonsurrell]

@dmsnell noticed that there's a remaining bug if < is followed by:

• [
• \
• ]
• ^
• _
• ` (backtick)

I'll work on a fix.

A fix was introduced to the Tag Processor to ensure that contiguous text in an HTML document emerges as a single text node spanning the full sequence. Unfortunately, that patch was marginally over-zealous in checking if a "<" started a syntax token or not. It used the following:

if ( 'A' <= $c && 'z' >= $c ) { ... }

This was based on the assumption that the A-Z and a-z letters are contiguous in the ASCII range; they aren't, and there's a gap of several characters in between. The result of this is that in some cases the parser created a text boundary when it didn't need to. Text boundaries can be surprising and can be created when reaching invalid syntax, HTML comments, and more hidden elements, so semantically this wasn't a major bug, but it was an aesthetic challenge.

In this patch the check is properly compared for both upper- and lower-case variants that could potentially form tag names.

if ( ( 'A' <= $c && 'Z' >= $c ) || ( 'a' <= $c && 'z' >= $c ) ) { ... }

This solves the problem and ensures that contiguous text appears as a single text node when scanning tokens.

Follow-up to [57489]
Props dmsnell, jonsurrell
See Core-60385

[dmsnell]

In 57542:

HTML API: Join text nodes on invalid-tag-name boundaries.

A fix was introduced to the Tag Processor to ensure that contiguous text
in an HTML document emerges as a single text node spanning the full
sequence. Unfortunately, that patch was marginally over-zealous in
checking if a "<" started a syntax token or not. It used the following:

<?php
if ( 'A' <= $c && 'z' >= $c ) { ... }

This was based on the assumption that the A-Z and a-z letters are
contiguous in the ASCII range; they aren't, and there's a gap of
several characters in between. The result of this is that in some
cases the parser created a text boundary when it didn't need to.
Text boundaries can be surprising and can be created when reaching
invalid syntax, HTML comments, and more hidden elements, so
semantically this wasn't a major bug, but it was an aesthetic
challenge.

In this patch the check is properly compared for both upper- and
lower-case variants that could potentially form tag names.

<?php
if ( ( 'A' <= $c && 'Z' >= $c ) || ( 'a' <= $c && 'z' >= $c ) ) { ... }

This solves the problem and ensures that contiguous text appears
as a single text node when scanning tokens.

Developed in ​https://github.com/WordPress/wordpress-develop/pull/6041
Discussed in https://core.trac.wordpress.org/ticket/60385

Follow-up to [57489]
Props dmsnell, jonsurrell
Fixes #60385

Merged in [57542]
​https://github.com/wordpress/wordpress-develop/commit/b7027a85e0ddf584bc707c5e5a686aa08bc13d52