Opened 6 weeks ago
Last modified 12 days ago
#66208 new enhancement
KSES: Re-implement using HTML API
| Reported by: | dmsnell | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | 7.2 |
| Component: | HTML API | Version: | 0.71 |
| Severity: | normal | Keywords: | has-patch has-unit-tests |
| Cc: | Focuses: |
Description (last modified by )
Since its initial commit, WordPress has relied on the KSES library to sanitize and normalize and transform content from untrusted sources. Unfortunately, the parsing inside of wp_kses() has never been spec-compliant, leading to numerous challenges in safely, efficiently, and reliably fulfilling its purpose.
The HTML API provides a mechanism for structurally interacting with HTML in an efficient manner, and wp_kses() should be rewritten to lean on that existing capability.
Resolves
- #25851 Large attribute values may crash PCRE patterns and cause content loss.
- #37698 Global pollution in
wp_kses_split()calls. - #48873 CSS contents are corrupted by
wp_kses(). - #51482
wp_kses()turns SCRIPT and STYLE content into renderable text. - #52333
wp_kses()and HTML disagree on the set of named character references. - #58377 Block names with consecutive hyphens are corrupted.
- #58921 Valid tag names are rejected from the allow-list.
- #59310
parse_blocks()called unnecessarily. - #61246
wp_kses()un-comments HTML comments. - #62024
wp_kses_post()incorrectly escapes “<” in attribute values.
Potentially resolves
Related
Change History (46)
This ticket was mentioned in ​PR #13271 on ​WordPress/wordpress-develop by ​@dmsnell.
6 weeks ago
#1
- Keywords has-patch has-unit-tests added
This ticket was mentioned in ​Slack in #core by dmsnell. ​View the logs.
5 weeks ago
This ticket was mentioned in ​Slack in #core by dmsnell. ​View the logs.
5 weeks ago
This ticket was mentioned in ​PR #14092 on ​WordPress/wordpress-develop by ​@jonsurrell.
5 weeks ago
#22
Make the Block Bindings unsafe HTML test pass whether KSES keeps or removes the text inside a removed SCRIPT element.
The 'unsafe HTML should be sanitized' case in data_different_get_value_callbacks() compared the rendered paragraph to an exact string. That string depends on how KSES handles the contents of a disallowed SCRIPT element, which changed in r64233 ("KSES: Reimplement with Tag Processor."). The test checks that the SCRIPT element is removed, not what happens to its text.
This moves the case into test_unsafe_html_from_source_is_sanitized(), which renders the bound paragraph and asserts with WP_HTML_Tag_Processor that the output contains no SCRIPT element.
This syncs with Gutenberg PR ​https://github.com/WordPress/gutenberg/pull/84294, which makes the same change to phpunit/block-bindings-test.php.
Use of AI Tools
AI assistance: Yes
Tool(s): Claude Code
Used for: The change was authored with Claude Code and reviewed by the author.
This ticket was mentioned in ​PR #14120 on ​WordPress/wordpress-develop by ​@dmsnell.
5 weeks ago
#23
Trac ticket: Core-66208
See wordpress/wordpress-develop#13271
When wp_kses() was refactored, it introduced block name normalization, but only for opening and void block delimiters whose attributes were changed during sanitization.
This left semantically-equivalent but mismatched delimiters.
This patch enforces normalization for all block names inside block delimiters.
​@dmsnell commented on ​PR #14120:
4 weeks ago
#25
Merged in r64233
7f6eac8a9f8cc5a5cb1113695010ca078773f745
This ticket was mentioned in ​PR #14136 on ​WordPress/wordpress-develop by ​@dmsnell.
4 weeks ago
#26
Trac ticket: Core-66208
When r64255 was added, normalizing all block delimiters inside wp_kses(), if a block contained an empty set of JSON attributes, but the JSON {} was present on the delimiter, then it would be reserialized as [] and misparse later.
This patch fixes that to only print the serialized attributes if there are any, implying a JSON object rather than an array.
Follow-up to: r64255
​@dmsnell commented on ​PR #14136:
4 weeks ago
#28
#29
@
4 weeks ago
A small downstream data point from bbPress, similar to the Core test adjusted in ​PR #14092: a formatting test added in ​bbPress r7954 expected legacy KSES output that retained alert(1) after removing its SCRIPT tags. The HTML API-backed parser now removes the content as intended, causing only a test failure and no bbPress runtime defect.
I adjusted the bbPress test in ​r7976. Exact sanitization and slashing checks now use an ordinary disallowed element, while SCRIPT coverage only requires its markup to be removed. It passes with both KSES parser modes and current WordPress trunk.
Thank you for the detailed report and legacy-parser filter. They made the cause straightforward to identify. đź’š
#30
@
4 weeks ago
BuddyPress passes slashed content to wp_kses(), both when ​saving and when ​reading stored values.
With [64233], this broke links in WordPress.org profile bios (​https://meta.trac.wordpress.org/ticket/8451). I pushed a hotfix to force the legacy parser there, until it's fixed in BuddyPress.
@espellcaste Do you think you could take a look at this when you get a chance?
#31
@
4 weeks ago
Thanks @obenland for noting this. I saw this turn up in a few places but also that the docs for wp_kses() have mentioned for some time “​This function expects unslashed data.”
I’m hoping this is a good opportunity to update code that was accidentally working in some cases before. There is a transform in the legacy implementation which only replaces a slashed double quote, but it wasn’t for “slashing” but made reference to some PCRE need (which, if it was actually required, was taking a blunt approach that prevented it from working generally, and by that I mean without breaking other existing inputs).
​@Mamaduka commented on ​PR #14092:
4 weeks ago
#32
I think this just needs to be committed. @sirreal, do you have time to handle this?
​@jonsurrell commented on ​PR #14092:
4 weeks ago
#34
Merged in r64355.
This ticket was mentioned in ​PR #14770 on ​WordPress/wordpress-develop by ​@salvatorecorsi.
2 weeks ago
#36
Description
The HTML API implementation of KSES decodes escaped square brackets in ordinary text nodes. This changes WordPress shortcode behavior: do_shortcode( wp_kses_post( '[example]' ) ) invokes a registered example shortcode instead of displaying its literal spelling. The same regression affects examples inside code and pre, and restricted authors encounter it through the normal post-save filters.
Preserve the distinction between literal brackets and brackets decoded from character references. Use the existing HTML decoder to identify reference-origin brackets in the original text token, then retain those as numeric references when serializing the already-normalized text. Literal shortcode syntax, other HTML character decoding, and existing newline/NULL handling continue through the existing paths. A compact byte-string map keeps the additional storage proportional to the number of brackets.
This is a focused follow-up to the escaped-bracket TODO in #13271. It covers ordinary #text nodes; HTML attributes and atomic RCDATA (title/textarea) need separate provenance handling and are outside this patch. It does not change shortcode parsing or claim that the entire KSES migration is complete.
Reproduction
- Register a harmless shortcode named
examplewhose callback returns a fixed marker. - Sanitize
[example] [example]withwp_kses_post()under the default WordPress bootstrap. - Pass the result through
do_shortcode().
Before: the callback runs twice. After: the encoded example remains literal and only the intentionally written shortcode runs. The regression tests also exercise wp_insert_post() as an author without unfiltered_html, then read and render the stored content.
Validation
Refreshed on October 23 after a normal merge of current trunk (3fb786ce), with unchanged source and test hashes. Native PHP 8.3 / PHPUnit 9.6, using the workspace's serialized, isolated test-database wrapper:
- Exact unchanged upstream source: 61 new test cases, 46 failures and 15 passing controls. All 61 cases pass after the correction (305 assertions).
- KSES and shortcode groups: 1,041 tests, 3,242 assertions, four skips, no failures. This scope includes the new tests.
- HTML decoder, modifiable text, input preprocessing, and post filtering classes: 572 tests, 1,383 assertions, one skip, no failures.
- Changed-file WordPress Coding Standards and PHP 7.4+ compatibility checks pass.
- Changed production file PHPStan analysis with the repository configuration passes.
- Full
npm run buildpasses; built KSES is byte-identical to the changed source.
The new static provider covers decimal, hexadecimal, named and alias references; leading zeroes; omitted semicolons; either encoded delimiter; mixed literal and escaped shortcodes; nested/unknown/non-bracket references; repeated sanitization; CRLF; PRE newlines; and NULL normalization. A separate test exercises real restricted-author post saving. The full Core suite and Docker E2E have not been run.
Use of AI Tools
AI assistance: Yes. Tool: OpenAI Codex. Used for investigation, implementation, regression tests, local validation, and independent automated code reviews. The account owner has not performed a human review of this patch. WordPress maintainer review is required.
​@dmsnell commented on ​PR #14770:
2 weeks ago
#37
@salvatorecorsi this is an issue we worked through when building the refactor. there are a lot of shaky ways to try and prevent the unescaping of square brackets, but ultimately it seemed best to leave it, for a number of reasons:
- above all, _shortcodes are escaped by doubling the brackets_ like
[[example]]. this is the canonical way to escape them. - there are code paths that need to process the unescaped HTML text, which might include URLs and
styleattribute values. if we intentionally send in character references after unescaping the text, we risk introducing bypasses to those other guards. - tracking the raw bytes through the transforms in
wp_kses()is fraught at best and impossible at worst. it’s fraught because we cannot confirm that a[from the raw text which appears in the output text in the same position _is the same[_; it could be coincidence that after transforming some prior content, that another intentional and unrelated bracket was put in there. - I’m worried about performance impacts of tracking this, though I haven’t measured to know if it’s a problem.
I considered something like you show for text nodes, because those don’t currently have any processing applied. as you scope-limited this to text nodes, that’s nice, but also sets up a weird expectation where _some_ escaped brackets are preserved and others aren’t.
by the way, I think it’s possible to speed up the detection of these inside text spans. while we might generally have some issues using a PCRE pattern to find character references, in specific cases we should be able to find comprehensive matches.
$bracket_pattern = <<<PCRE ~ \A # immediately following the found & (?: (?:[lr](sqb|brack)) # named character references | #(?: x0*5[BD] # hex numeric | 0*9[13] # decimal numeric ) ) \K # discard the results/don’t allocate ~ix PCRE; while ( $at < $end ) { $next_at = strcspn( $raw_text, '&', $at, $end_of_span ); … // Filter on candidate matches. if ( 1 !== preg_match( $bracket_pattern, $raw_text, $match, PREG_OFFSET_CAPTURE, $at ) ) { $at = $next_at + 1; continue; } // rely on the real decoder just for these, because the semicolon rules are complicated. $reference = WP_HTML_Decoder::read_character_reference( … ); … }
anyway, that’s just a thought. it could speed up your approach, or it may not. the code I shared is probably terribly buggy, but it’s meant to convey the idea only.
---
for attribute values this breaks down hard because we expect them to be transformed. unless we re-engineered wp_kses() transforms (such as URL and style sanitization), I don’t see how we can track the brackets from the start to the end. what have you considered?
​@salvatorecorsi commented on ​PR #14770:
2 weeks ago
#38
@dmsnell Thanks for explaining the earlier reasoning. I've investigated this further with Codex-assisted analysis and automated checks, including a comparison against the legacy path and a small performance benchmark.
The compatibility case I think is worth preserving is existing entity-encoded text. With a registered probe shortcode:
| Input | Current legacy KSES | HTML API before this patch | With this patch |
|---|---|---|---|
[probe] | No callback | Callback executes | No callback |
[probe] | Callback executes | Callback executes | Callback executes |
[[probe]] | No callback | No callback | No callback |
I agree that doubled brackets are the documented shortcode escape. My concern is that they do not retrofit existing encoded examples, which can start executing after a restricted author's save. They also depend on the shortcode being registered: [[unknown]] retains both brackets when it is unregistered. The 61 focused tests still pass (305 assertions), including the restricted-author save case.
For ordinary text, the mapping uses bracket order, not byte positions. The current newline/NUL normalization and HTML escaping preserve that order; the original token and the same HTML decoder identify which brackets came from references. This is the specific invariant the patch relies on. It does not restore raw HTML or change attribute, URL, or CSS sanitization.
I agree that this reasoning cannot simply be extended to attributes: their filtering can remove or rewrite substrings. I do not have a sound general provenance solution for those transforms, and would not reinsert raw attribute values after sanitization. The text-only behavior is consequently partial. Allowed title/textarea remain outside scope, and further probes confirmed other unresolved normalization cases even in ordinary text: [probe] can still become an active shortcode, and " inside a quoted shortcode parameter can change argument parsing. These also occur before this patch; preserving brackets alone does not solve the entire compatibility problem.
Your performance concern is justified. On PHP 8.3.28 CLI, comparing the exact base and patched KSES sources with warmed, repeated measurements, this synthetic input:
'<p>[ ' . str_repeat( '& © © — ', 500 ) . '</p>'
takes roughly 3.43 ms before versus 6.89 ms after, with identical output. One literal bracket triggers an unnecessary second decode of all those unrelated entities. Ordinary 9 KB prose measured about 0.71 versus 0.73 ms; these are local function microbenchmarks, not request-level timings.
I would first address that avoidable work using a cheap candidate filter followed by the real decoder, as you suggested, and measure it again. Would you consider this limited ordinary-text compatibility fix worthwhile after reducing that cost, or would you prefer a broader approach first?
​@dmsnell commented on ​PR #14770:
2 weeks ago
#39
@salvatorecorsi there’s a lot in your response I unfortunately find hard to read through and understand. but here is perhaps an incomplete and unsatisfactory response.
whether we could preserve these in raw text nodes is something we knew we could do. I’ve explored ​other text node normalization as well. what I’m most worried about in doing this is presenting a situation where _some_ shortcodes are preserved and others aren’t — one aspect of the wp_kses() rewrite that was important to me personally was predictability. the legacy code’s behavior was highly unpredictable because it directly depended on what syntax was presented in its inputs. the rewrite should be entirely regular, presenting developers an interface they can more or less intuit and rely on (without having to remember a list of exceptions to its behavior).
I appreciate that in your compatibility chart, you note that this is a divergence in behavior for how the old KSES worked. However, I don’t think it tells the full story. It’s an important question, and I want you to know I’m thinking through this with an open mind (and my rambling is characteristic of me trying to share my thought process transparently).
---
there may be opportunities to explore this at the end, using your approach with counting brackets, but instead counting actual shortcodes. that might even work for attribute values, except for some situations. one glaring example are CSS selectors which might appear, for example, in data- attributes.
<div data-target="[hidden]">
That’s not a Shortcode, though [ and ] are 100% benign in those cases.
There is vague suggestion that Shortcodes may only appear in text nodes and inside attribute values. That’s vague because the shortcode parser extends beyond that. What do you think about exploring this idea:
- immediately before each text node or attribute value flushes into the output buffer
- detect contained shortcode syntax (as you noted, this is not a totally tractable problem due to the way that the shortcode parser currently works, with registered or sought-after shortcodes being passed to the parser)
- if those shortcodes weren’t present in the raw input for that text node or attribute value, then escape the brackets
​@salvatorecorsi commented on ​PR #14770:
2 weeks ago
#40
That would give text and attributes the same rule: normalization must not introduce a shortcode that the raw input didn't contain. I'll prototype the comparison at those flush points, including encoded tag names and CSS-selector attributes. The registration-dependent parser is the part I still need to work through.
​@salvatorecorsi commented on ​PR #14770:
2 weeks ago
#41
I tried this in an isolated prototype. It catches encoded names in both text and attributes, and bracket escaping leaves the decoded CSS selector unchanged. I'd keep detection independent of registered tags so plugin activation doesn't change the saved result.
Counts still lose which occurrence was literal: ENCODED: [probe] LITERAL: [probe] can keep the first active and escape the second. Matching arguments doesn't resolve identical occurrences; preserving the literal one needs a link to its raw source.
There's also [pro<notallowed>be]: removing that disallowed tag creates [probe] across two text flushes. Should this PR cover those cases too, or specifically prevent new syntax within each individual text/attribute value?
This ticket was mentioned in ​PR #14841 on ​WordPress/wordpress-develop by ​@nofearinc.
2 weeks ago
#42
The HTML API based wp_kses() stops sanitizing at the first NOSCRIPT opening tag (break in sanitize()), so everything after it is dropped. For users without unfiltered_html (Authors, Contributors, commenters), a post or comment that contains a NOSCRIPT fallback loses the rest of its content on save:
| Input | 7.1.3 | 7.2-beta1 |
|---|---|---|
Before <noscript>[[Image(a.jpg)]]</noscript> after the noscript. | Before [[Image(a.jpg)]] after the noscript. | Before
|
This skips the NOSCRIPT element together with its contents and continues after its closer. Nothing inside a NOSCRIPT is emitted, so untrusted input still cannot produce one, including through a </noscript> hidden in an attribute value (one of the test cases). An unclosed NOSCRIPT still truncates, like other incomplete input.
Testing
vendor/bin/phpunit --filter test_wp_kses_removes_noscript_and_keeps_following_content: 5 of the 6 cases fail without the patch, all pass with it.vendor/bin/phpunit --group kses(529 tests) and--group html-api(2051 tests) pass.- Merges cleanly with #14770.
Use of AI Tools
AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5.5
Used for: finding the issue, drafting the fix and tests, and running them. I reviewed every line and reproduced the output above on 7.1.3 and 7.2-beta1.
---
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See ​GitHub Pull Requests for Code Review in the Core Handbook for more details.
🤖 Generated with ​Claude Code
#43
@
2 weeks ago
In 7.2-beta1, wp_kses() stops sanitizing at the first NOSCRIPT opening tag (break in sanitize()), so for users without unfiltered_html everything after it is lost on save. Before <noscript><img src="a.jpg"></noscript> after the noscript. becomes Before , where 7.1.3 kept Before <img src="a.jpg"> after the noscript.
​https://github.com/WordPress/wordpress-develop/pull/14841 skips the NOSCRIPT element together with its contents and continues after its closer, so untrusted input still cannot produce one. An unclosed NOSCRIPT still truncates, in line with how incomplete input is handled since PR 13271. @dmsnell, was the break meant as a fail-closed choice? Happy to adjust.
AI assistance: Claude Code (Claude Opus 5.5) for finding the case and drafting the patch and tests; reviewed and tested by me.
#44
@
13 days ago
@nofearinc the reason NOSCRIPT is rejected is because of the different parsing rules for scription-mode-enabled and scripting-mode-disabled parsing. there is one potential way to allow this, which I hadn’t considered before: upon finding a NOSCRIPT opening tag, find the scripting-mode-disabled NOSCRIPT closing tag and note it’s position. then, continue parsing until that position, and if the next token is anything other than the NOSCRIPT closing tag, reject to before it was opened.
the core issue is that in a script-mode-disabled environment like Core, there actually exist tags inside NOSCRIPT. in a script-mode-enabled environment like a browser with JS enabled, there are _no_ tags inside NOSCRIPT, meaning that the allowed HTML rules don’t apply, and parsing forward tag-by-tag could lead to unexpected breakout from the NOSCRIPT element.
<noscript><a title="</noscript>">oops</a></noscript>
in a script-mode-enabled environment there is no A element in the above snippet: the NOSCRIPT ends at the first </noscript>, and what follows is ">oops, an ignored unexpected </a>, and an ignored unexpected </noscript>.
in a script-mode-disabled environment there is an A element containing a title attribute whose value is </noscript>, whose contents are oops, and then nothing else.
so there’s no actual way to apply one set of rules against NOSCRIPT. given that browsers hide the NOSCRIPT contents with script-mode-enabled, it could make sense to clobber the contents, applying the script-mode-disabled parsing only. however, that relies on a private function skip_rawtext() which we weren’t sure we could expose properly for 7.2.
it is possible to match the NOSCRIPT closer in script-mode-enabled with a single PCRE pattern. ​using my PCRE-based HTML segmenter as an example.
<?php if ( 'NOSCRIPT' === $token_name && ! $is_closer ) { $ws = " \t\f\r\n"; $avalue = "(?:\"[^\"]*\"|'[^']*'|(?![\"'])[^>{$ws}]*)"; $after_aname = "(?:[{$ws}]*+=[{$ws}]*+{$avalue}|(?![{$ws}]*+=))"; $pattern = "~</?noscript()(?:[{$ws}/]*(?:[^=/>{$ws}]+{$after_aname}|=[^=/>{$ws}]*{$after_aname}))*[{$ws}/]*>\K~i"; if ( 1 === preg_match( $pattern, $this->html, $closer_match, PREG_OFFSET_CAPTURE, $here->start + $here->length ) { $noscript_end_at = $closer_match[0][1]; $this->set_bookmark( 'inside_noscript' ); while ( $this->next_token() ) { if ( $this->get_span()->start < $noscript_end_at ) { continue; } if ( $this->get_span()->start > $noscript_end_at || 'NOSCRIPT' !== $this->get_tag() || ! $this->is_tag_closer() ) { /* * Only allow NOSCRIPT if the script-mode-enabled * and script-mode-disabled boundaries align. */ break 2; } break; } $this->seek( 'inside_noscript' ); } }
This is crude, and the code is there for concept only. In a real implementation we would want to avoid the seek and instead track state as being inside NOSCRIPT, and storing $before_noscript, processing once, and then truncating if the two don’t align (see SVG/MathML handling for an example).
It would also need to remove the entire NOSCRIPT contents if NOSCRIPT weren’t allowed.
Feel free to work on this. I hope this conveys the rationale for not entering into it. We can always make wp_kses() more robust, but in this rewrite, we want to be cautious and certain before we allow HTML patterns we can’t fully verify.
​@dmsnell commented on ​PR #14770:
13 days ago
#45
I'd keep detection independent of registered tags so plugin activation doesn't change the saved result.
this is an easy escape hatch, and something I plan on proposing, but it’s a backwards-compatibility break with unknown consequences. shortcode detection is kind of a wild mess.
removing that disallowed tag creates
[probe]across two text flushes
you are starting to see the complicated cases this touches. another change to shortcodes would free this up, and I think the _intent_ is that shortcodes today only exist within complete text nodes, or inside HTML attributes, but the current shortcode parser doesn’t do that.
---
and this is all work meant to ensure the prevention of creating shortcodes, yet we have to deal with representing non-shortcode lookalikes.
a good metric might simply be that we create nothing that could parse as a shortcode through wp_kses() that wasn’t already there. it’s not right to escape shortcodes by using the escaped brackets (because this is how CSS selectors, for example, need to be represented); just as it’s not right to escape everything that looks like a shortcode using double brackets (because things like CSS selectors are not shortcodes, so shouldn’t be turned into them).
this all brings me back to shortcodes, where the heart of the problem is. we’re supposed to escape anything that could appear like a shortcode, meaning that if we ever expect data-selector="[hidden]", we should be writing it like data-selector="[[hidden]]", except then we have to rely on shortcode processing that content before it gets sent to a browser, or to an API response, which often doesn’t happen.
​@salvatorecorsi commented on ​PR #14770:
12 days ago
#46
You're right that registration-independent detection would change compatibility, so I won't build that assumption into this patch. The prototype only checked the decoded selector value; it didn't establish that every shortcode-processing path remains compatible. I'll keep the implementation unchanged for now, until there's a rule that prevents newly created shortcode syntax without rewriting literal selector values, including when removing markup joins text across tokens.
![(please configure the [header_logo] section in trac.ini)](/chrome/site/your_project_logo.png)
Trac ticket: Core-66208
Trac ticket: Core-65984
Replaces #6577
Description
Rewrites
wp_kses()to rely on the HTML API for structural and reliable application of sanitization rules, normalizing the output for improved downstream parsing.Notables
wp_kses_force_legacy_parserprovides the choice of whether to use this new parser or stick with the legacy code.Fixes
wp_kses()un-comments HTML comments.wp_kses_post()incorrectly escapes "<" attributes values.Todo
~Merge after #13273, which accounts for three of the failing tests.~
wp_kses(), it’s possible to simply wait until an opened element is closed based on depth, and skip that closing element if it exists.wp_kses()with intentionally-incomplete input, for example, a wrapper opening tag with part of the content, separately from the closer. closing open elements does a good job of isolating content, but legacy behaviors depend too much on the more procedural use ofwp_kses()so isolation cannot be reasonably added without mangling websites.pre_ksesfilters but then callpre_ksesNotes