HTML API: Unclosed "funky comment" may not pause at incomplete token

[jonsurrell]

The HTML Tag Processor may report ::paused_at_incomplete_token() === true correctly when the HTML ends in an open "funky comment."

The HTML </# should be handled the same as </# , in both cases a funky comment has been opened but no closer is found before the HTML document ends. Both cases should both be treated as incomplete input, however the former case (with no trailing space) does not report an incomplete token.

[jonsurrell]

I've confirmed this in 6.5, but the behavior may have existed earlier.

• Fix an issue where </# is not treated as incomplete input but </# is. Both should be incomplete input.
• Add a test for the issue.

---

The HTML Tag Processor may report ::paused_at_incomplete_token() === true correctly when the HTML ends in an open "funky comment."

The HTML </# should be handled the same as </# , in both cases a funky comment has been opened but no closer is found before the HTML document ends. Both cases should both be treated as incomplete input, however the former case (with no trailing space) does not report an incomplete token.

Trac ticket: https://core.trac.wordpress.org/ticket/61831

[jonsurrell]

This was discovered by the HTML5lib-tests full HTML API parser tests, in progress in ​https://github.com/WordPress/wordpress-develop/pull/7117.

Trac ticket: Core-61646

Previously, the html5lib tests have only run in the fragment parser mode, but in #6977 a full parser was introduced to the HTML API. In this patch, the html5lib tests are running in the full parser mode where applicable.

#7146 is merged here and should be landed first.

- Tests: 609, Assertions: 405, Skipped: 204.
+ Tests: 1498, Assertions: 787, Skipped: 711.

This vastly increases the tests that are used from HTML5lib-tests from 609 to 1498! It's helped to discover two issues:

• https://core.trac.wordpress.org/ticket/61831
• https://core.trac.wordpress.org/ticket/61810

[dmsnell]

In 58858:

HTML API: Truncated funky comments should cause the Tag Processor to pause.

A state change was missing in the Tag Processor when the input is too short to
find a comment closer after an opened funky comment. This patch fixes a issue
where </# does not correctly report incomplete input, but </# does.

Developed in ​https://github.com/wordpress/wordpress-develop/pull/7146
Discussed in https://core.trac.wordpress.org/ticket/61831

Props: jonsurrell.
Fixes #61831.

Merged in [58858]
​https://github.com/wordpress/wordpress-develop/commit/534b830b371644cdd177f79317b9d28b0dcbbc85

Merged in [58859]
​https://github.com/wordpress/wordpress-develop/commit/9928cd6bcb35d14a700e3930355b6703f0cb4dc8