Make WordPress Core

Opened 6 weeks ago

Last modified 12 days ago

#66208 new enhancement

KSES: Re-implement using HTML API

Reported by: dmsnell Owned by:
Priority: normal Milestone: 7.2
Component: HTML API Version: 0.71
Severity: normal Keywords: has-patch has-unit-tests
Cc: Focuses:

Description (last modified by dmsnell)

Since its initial commit, WordPress has relied on the KSES library to sanitize and normalize and transform content from untrusted sources. Unfortunately, the parsing inside of wp_kses() has never been spec-compliant, leading to numerous challenges in safely, efficiently, and reliably fulfilling its purpose.

The HTML API provides a mechanism for structurally interacting with HTML in an efficient manner, and wp_kses() should be rewritten to lean on that existing capability.

Resolves

  • #25851 Large attribute values may crash PCRE patterns and cause content loss.
  • #37698 Global pollution in wp_kses_split() calls.
  • #48873 CSS contents are corrupted by wp_kses().
  • #51482 wp_kses() turns SCRIPT and STYLE content into renderable text.
  • #52333 wp_kses() and HTML disagree on the set of named character references.
  • #58377 Block names with consecutive hyphens are corrupted.
  • #58921 Valid tag names are rejected from the allow-list.
  • #59310 parse_blocks() called unnecessarily.
  • #61246 wp_kses() un-comments HTML comments.
  • #62024 wp_kses_post() incorrectly escapes “<” in attribute values.

Potentially resolves

  • #51093 CSS > corruption.
  • #56118 Slashes are improperly stripped when exporting data.
  • #63881 Deprecate wp_kses_stripslashes().
  • #65599 HTML sanitization API implementation.

Change History (46)

This ticket was mentioned in ​PR #13271 on ​WordPress/wordpress-develop by ​@dmsnell.


6 weeks ago
#1

  • Keywords has-patch has-unit-tests added

Trac ticket: Core-66208
Trac ticket: Core-65984

Replaces #6577

Description

Rewrites wp_kses() to rely on the HTML API for structural and reliable application of sanitization rules, normalizing the output for improved downstream parsing.

Notables

  • A new filter wp_kses_force_legacy_parser provides the choice of whether to use this new parser or stick with the legacy code.

Fixes

  • Core-61246 wp_kses() un-comments HTML comments.
  • Core-62024 wp_kses_post() incorrectly escapes "<" attributes values.

Todo

~Merge after #13273, which accounts for three of the failing tests.~

  • [x] self-closing non-HTML elements
  • [~] remove opening tag when required attributes are missing, and closing tag
    • while this would be a nice enhancement it’s going to be left out of this work to preserve existing behaviors. with the HTML Processor powering wp_kses(), it’s possible to simply wait until an opened element is closed based on depth, and skip that closing element if it exists.
  • [x] replace C0 controls
    • [x] replace C0 controls with their escapes, rather than stripping them away
    • [x] replace C0 controls in attribute values?
    • [ ] original commit _removing C0_ controls is c7fd8c7973
    • [ ] C0 controls are left in-place to prevent problems with creating new syntax through their removal
  • [~] handle incomplete parsing, including closing all open elements
    • plenty of existing code in Core calls wp_kses() with intentionally-incomplete input, for example, a wrapper opening tag with part of the content, separately from the closer. closing open elements does a good job of isolating content, but legacy behaviors depend too much on the more procedural use of wp_kses() so isolation cannot be reasonably added without mangling websites.
  • [x] if SVG or MATH are not allowed, the entire element should disappear
  • [x] remove default pre_kses filters but then call pre_kses

Notes

#2 @dmsnell
6 weeks ago

#61246 was marked as a duplicate.

#3 @dmsnell
6 weeks ago

  • Description modified (diff)

#4 @dmsnell
6 weeks ago

#62024 was marked as a duplicate.

#5 @dmsnell
6 weeks ago

#25851 was marked as a duplicate.

#6 @dmsnell
6 weeks ago

  • Description modified (diff)

#7 @dmsnell
6 weeks ago

#52333 was marked as a duplicate.

#8 @dmsnell
6 weeks ago

  • Description modified (diff)

#9 @dmsnell
6 weeks ago

#58377 was marked as a duplicate.

#10 @dmsnell
6 weeks ago

  • Description modified (diff)

#11 @dmsnell
6 weeks ago

  • Description modified (diff)

#12 @dmsnell
6 weeks ago

  • Description modified (diff)

#13 @dmsnell
6 weeks ago

#48873 was marked as a duplicate.

#14 @dmsnell
6 weeks ago

  • Description modified (diff)

#15 @dmsnell
6 weeks ago

  • Description modified (diff)

#16 @dmsnell
6 weeks ago

#51482 was marked as a duplicate.

#17 @dmsnell
6 weeks ago

  • Description modified (diff)

#18 @dmsnell
6 weeks ago

  • Description modified (diff)

#19 @dmsnell
6 weeks ago

  • Description modified (diff)

This ticket was mentioned in ​Slack in #core by dmsnell. ​View the logs.


5 weeks ago

This ticket was mentioned in ​Slack in #core by dmsnell. ​View the logs.


5 weeks ago

This ticket was mentioned in ​PR #14092 on ​WordPress/wordpress-develop by ​@jonsurrell.


5 weeks ago
#22

Make the Block Bindings unsafe HTML test pass whether KSES keeps or removes the text inside a removed SCRIPT element.

The 'unsafe HTML should be sanitized' case in data_different_get_value_callbacks() compared the rendered paragraph to an exact string. That string depends on how KSES handles the contents of a disallowed SCRIPT element, which changed in r64233 ("KSES: Reimplement with Tag Processor."). The test checks that the SCRIPT element is removed, not what happens to its text.

This moves the case into test_unsafe_html_from_source_is_sanitized(), which renders the bound paragraph and asserts with WP_HTML_Tag_Processor that the output contains no SCRIPT element.

This syncs with Gutenberg PR ​https://github.com/WordPress/gutenberg/pull/84294, which makes the same change to phpunit/block-bindings-test.php.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Used for: The change was authored with Claude Code and reviewed by the author.

This ticket was mentioned in ​PR #14120 on ​WordPress/wordpress-develop by ​@dmsnell.


5 weeks ago
#23

Trac ticket: Core-66208

See wordpress/wordpress-develop#13271

When wp_kses() was refactored, it introduced block name normalization, but only for opening and void block delimiters whose attributes were changed during sanitization.

This left semantically-equivalent but mismatched delimiters.

This patch enforces normalization for all block names inside block delimiters.

#24 @dmsnell
4 weeks ago

In 64255:

KSES: Normalize all block delimiters.

When wp_kses() was refactored, it introduced block name normalization, but only for opening and void block delimiters whose attributes were changed during sanitization.

This left semantically-equivalent but mismatched delimiters.

This patch enforces full normalization for all block delimiters.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/14120
Discussed in: https://core.trac.wordpress.org/ticket/66208

Follow-up to: r64233.

Props dmsnell.
See #66208.

​@dmsnell commented on ​PR #14120:


4 weeks ago
#25

Merged in r64233
7f6eac8a9f8cc5a5cb1113695010ca078773f745

This ticket was mentioned in ​PR #14136 on ​WordPress/wordpress-develop by ​@dmsnell.


4 weeks ago
#26

Trac ticket: Core-66208

When r64255 was added, normalizing all block delimiters inside wp_kses(), if a block contained an empty set of JSON attributes, but the JSON {} was present on the delimiter, then it would be reserialized as [] and misparse later.

This patch fixes that to only print the serialized attributes if there are any, implying a JSON object rather than an array.

Follow-up to: r64255

#27 @dmsnell
4 weeks ago

In 64258:

KSES: Only serialize non-empty block JSON attributes.

In r64255 wp_kses() completely reserializes every block delimiter, including any JSON attributes. Unfortunately, if a delimiter contained no JSON attributes, but still presented an empty JSON map, these attributes would be re-serialized as an empty list due to the ambiguity of empty PHP arrays. This led to blocks no-longer parsing as blocks.

wp_kses( '<!-- wp:block {} -->' ) turns into <!-- wp:block [] -->

This patch checks if any attributes exist before serializing them into the block delimiter. There is a pre-existing version of this problem in WordPress for block JSON attributes whose value is an empty map, but r64255 did not change how those are modified through wp_kses().

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/14136
Discussed in: https://core.trac.wordpress.org/ticket/66208

Follow-up to: r64255.

Props bjorsch, dmsnell.
See #66208.

#29 @johnjamesjacoby
4 weeks ago

A small downstream data point from bbPress, similar to the Core test adjusted in ​PR #14092: a formatting test added in ​bbPress r7954 expected legacy KSES output that retained alert(1) after removing its SCRIPT tags. The HTML API-backed parser now removes the content as intended, causing only a test failure and no bbPress runtime defect.

I adjusted the bbPress test in ​r7976. Exact sanitization and slashing checks now use an ordinary disallowed element, while SCRIPT coverage only requires its markup to be removed. It passes with both KSES parser modes and current WordPress trunk.

Thank you for the detailed report and legacy-parser filter. They made the cause straightforward to identify. đź’š

#30 @obenland
4 weeks ago

BuddyPress passes slashed content to wp_kses(), both when ​saving and when ​reading stored values.

With [64233], this broke links in WordPress.org profile bios (​https://meta.trac.wordpress.org/ticket/8451). I pushed a hotfix to force the legacy parser there, until it's fixed in BuddyPress.

@espellcaste Do you think you could take a look at this when you get a chance?

#31 @dmsnell
4 weeks ago

Thanks @obenland for noting this. I saw this turn up in a few places but also that the docs for wp_kses() have mentioned for some time “​This function expects unslashed data.”

I’m hoping this is a good opportunity to update code that was accidentally working in some cases before. There is a transform in the legacy implementation which only replaces a slashed double quote, but it wasn’t for “slashing” but made reference to some PCRE need (which, if it was actually required, was taking a blunt approach that prevented it from working generally, and by that I mean without breaking other existing inputs).

​@Mamaduka commented on ​PR #14092:


4 weeks ago
#32

I think this just needs to be committed. @sirreal, do you have time to handle this?

#33 @jonsurrell
4 weeks ago

In 64355:

Tests: Fix Block Bindings unsafe HTML test.

Assert that the output has no SCRIPT tag, without depending on how KSES removes it.

Developed in: ​https://github.com/WordPress/wordpress-develop/pull/14092

Follow-up to r64233.

Props jonsurrell, mamaduka, andrewserong.
See #66208.

​@jonsurrell commented on ​PR #14092:


4 weeks ago
#34

Merged in r64355.

#35 @espellcaste
3 weeks ago

@obenland Tks for the ping. I'm looking into it for BuddyPress.

This ticket was mentioned in ​PR #14770 on ​WordPress/wordpress-develop by ​@salvatorecorsi.


2 weeks ago
#36

Description

The HTML API implementation of KSES decodes escaped square brackets in ordinary text nodes. This changes WordPress shortcode behavior: do_shortcode( wp_kses_post( '&#91;example&#93;' ) ) invokes a registered example shortcode instead of displaying its literal spelling. The same regression affects examples inside code and pre, and restricted authors encounter it through the normal post-save filters.

Preserve the distinction between literal brackets and brackets decoded from character references. Use the existing HTML decoder to identify reference-origin brackets in the original text token, then retain those as numeric references when serializing the already-normalized text. Literal shortcode syntax, other HTML character decoding, and existing newline/NULL handling continue through the existing paths. A compact byte-string map keeps the additional storage proportional to the number of brackets.

This is a focused follow-up to the escaped-bracket TODO in #13271. It covers ordinary #text nodes; HTML attributes and atomic RCDATA (title/textarea) need separate provenance handling and are outside this patch. It does not change shortcode parsing or claim that the entire KSES migration is complete.

Reproduction

  1. Register a harmless shortcode named example whose callback returns a fixed marker.
  2. Sanitize &#91;example&#93; [example] with wp_kses_post() under the default WordPress bootstrap.
  3. Pass the result through do_shortcode().

Before: the callback runs twice. After: the encoded example remains literal and only the intentionally written shortcode runs. The regression tests also exercise wp_insert_post() as an author without unfiltered_html, then read and render the stored content.

Validation

Refreshed on October 23 after a normal merge of current trunk (3fb786ce), with unchanged source and test hashes. Native PHP 8.3 / PHPUnit 9.6, using the workspace's serialized, isolated test-database wrapper:

  • Exact unchanged upstream source: 61 new test cases, 46 failures and 15 passing controls. All 61 cases pass after the correction (305 assertions).
  • KSES and shortcode groups: 1,041 tests, 3,242 assertions, four skips, no failures. This scope includes the new tests.
  • HTML decoder, modifiable text, input preprocessing, and post filtering classes: 572 tests, 1,383 assertions, one skip, no failures.
  • Changed-file WordPress Coding Standards and PHP 7.4+ compatibility checks pass.
  • Changed production file PHPStan analysis with the repository configuration passes.
  • Full npm run build passes; built KSES is byte-identical to the changed source.

The new static provider covers decimal, hexadecimal, named and alias references; leading zeroes; omitted semicolons; either encoded delimiter; mixed literal and escaped shortcodes; nested/unknown/non-bracket references; repeated sanitization; CRLF; PRE newlines; and NULL normalization. A separate test exercises real restricted-author post saving. The full Core suite and Docker E2E have not been run.

Use of AI Tools

AI assistance: Yes. Tool: OpenAI Codex. Used for investigation, implementation, regression tests, local validation, and independent automated code reviews. The account owner has not performed a human review of this patch. WordPress maintainer review is required.

​@dmsnell commented on ​PR #14770:


2 weeks ago
#37

@salvatorecorsi this is an issue we worked through when building the refactor. there are a lot of shaky ways to try and prevent the unescaping of square brackets, but ultimately it seemed best to leave it, for a number of reasons:

  • above all, _shortcodes are escaped by doubling the brackets_ like [[example]]. this is the canonical way to escape them.
  • there are code paths that need to process the unescaped HTML text, which might include URLs and style attribute values. if we intentionally send in character references after unescaping the text, we risk introducing bypasses to those other guards.
  • tracking the raw bytes through the transforms in wp_kses() is fraught at best and impossible at worst. it’s fraught because we cannot confirm that a [ from the raw text which appears in the output text in the same position _is the same [_; it could be coincidence that after transforming some prior content, that another intentional and unrelated bracket was put in there.
  • I’m worried about performance impacts of tracking this, though I haven’t measured to know if it’s a problem.

I considered something like you show for text nodes, because those don’t currently have any processing applied. as you scope-limited this to text nodes, that’s nice, but also sets up a weird expectation where _some_ escaped brackets are preserved and others aren’t.

by the way, I think it’s possible to speed up the detection of these inside text spans. while we might generally have some issues using a PCRE pattern to find character references, in specific cases we should be able to find comprehensive matches.

$bracket_pattern = <<<PCRE
~
        \A                          # immediately following the found &
        (?:
                (?:[lr](sqb|brack)) # named character references
                |
                #(?:
                        x0*5[BD]    # hex numeric
                        |
                        0*9[13]     # decimal numeric
                )
        )
        \K                          # discard the results/don’t allocate
~ix
PCRE;


while ( $at < $end ) {
        $next_at = strcspn( $raw_text, '&', $at, $end_of_span );
        …

        // Filter on candidate matches.
        if ( 1 !== preg_match( $bracket_pattern, $raw_text, $match, PREG_OFFSET_CAPTURE, $at ) ) {
                $at = $next_at + 1;
                continue;
        }

        // rely on the real decoder just for these, because the semicolon rules are complicated.
        $reference = WP_HTML_Decoder::read_character_reference( … );

        …
}

anyway, that’s just a thought. it could speed up your approach, or it may not. the code I shared is probably terribly buggy, but it’s meant to convey the idea only.

---

for attribute values this breaks down hard because we expect them to be transformed. unless we re-engineered wp_kses() transforms (such as URL and style sanitization), I don’t see how we can track the brackets from the start to the end. what have you considered?

​@salvatorecorsi commented on ​PR #14770:


2 weeks ago
#38

@dmsnell Thanks for explaining the earlier reasoning. I've investigated this further with Codex-assisted analysis and automated checks, including a comparison against the legacy path and a small performance benchmark.

The compatibility case I think is worth preserving is existing entity-encoded text. With a registered probe shortcode:

Input Current legacy KSES HTML API before this patch With this patch
&#91;probe&#93; No callback Callback executes No callback
[probe] Callback executes Callback executes Callback executes
[[probe]] No callback No callback No callback

I agree that doubled brackets are the documented shortcode escape. My concern is that they do not retrofit existing encoded examples, which can start executing after a restricted author's save. They also depend on the shortcode being registered: [[unknown]] retains both brackets when it is unregistered. The 61 focused tests still pass (305 assertions), including the restricted-author save case.

For ordinary text, the mapping uses bracket order, not byte positions. The current newline/NUL normalization and HTML escaping preserve that order; the original token and the same HTML decoder identify which brackets came from references. This is the specific invariant the patch relies on. It does not restore raw HTML or change attribute, URL, or CSS sanitization.

I agree that this reasoning cannot simply be extended to attributes: their filtering can remove or rewrite substrings. I do not have a sound general provenance solution for those transforms, and would not reinsert raw attribute values after sanitization. The text-only behavior is consequently partial. Allowed title/textarea remain outside scope, and further probes confirmed other unresolved normalization cases even in ordinary text: [pr&#111;be] can still become an active shortcode, and &quot; inside a quoted shortcode parameter can change argument parsing. These also occur before this patch; preserving brackets alone does not solve the entire compatibility problem.

Your performance concern is justified. On PHP 8.3.28 CLI, comparing the exact base and patched KSES sources with warmed, repeated measurements, this synthetic input:

'<p>[ ' . str_repeat( '&amp; &copy; &#169; &mdash; ', 500 ) . '</p>'

takes roughly 3.43 ms before versus 6.89 ms after, with identical output. One literal bracket triggers an unnecessary second decode of all those unrelated entities. Ordinary 9 KB prose measured about 0.71 versus 0.73 ms; these are local function microbenchmarks, not request-level timings.

I would first address that avoidable work using a cheap candidate filter followed by the real decoder, as you suggested, and measure it again. Would you consider this limited ordinary-text compatibility fix worthwhile after reducing that cost, or would you prefer a broader approach first?

​@dmsnell commented on ​PR #14770:


2 weeks ago
#39

@salvatorecorsi there’s a lot in your response I unfortunately find hard to read through and understand. but here is perhaps an incomplete and unsatisfactory response.

whether we could preserve these in raw text nodes is something we knew we could do. I’ve explored ​other text node normalization as well. what I’m most worried about in doing this is presenting a situation where _some_ shortcodes are preserved and others aren’t — one aspect of the wp_kses() rewrite that was important to me personally was predictability. the legacy code’s behavior was highly unpredictable because it directly depended on what syntax was presented in its inputs. the rewrite should be entirely regular, presenting developers an interface they can more or less intuit and rely on (without having to remember a list of exceptions to its behavior).

I appreciate that in your compatibility chart, you note that this is a divergence in behavior for how the old KSES worked. However, I don’t think it tells the full story. It’s an important question, and I want you to know I’m thinking through this with an open mind (and my rambling is characteristic of me trying to share my thought process transparently).

---

there may be opportunities to explore this at the end, using your approach with counting brackets, but instead counting actual shortcodes. that might even work for attribute values, except for some situations. one glaring example are CSS selectors which might appear, for example, in data- attributes.

<div data-target="[hidden]">

That’s not a Shortcode, though &#x5B; and &#x5D; are 100% benign in those cases.

There is vague suggestion that Shortcodes may only appear in text nodes and inside attribute values. That’s vague because the shortcode parser extends beyond that. What do you think about exploring this idea:

  • immediately before each text node or attribute value flushes into the output buffer
  • detect contained shortcode syntax (as you noted, this is not a totally tractable problem due to the way that the shortcode parser currently works, with registered or sought-after shortcodes being passed to the parser)
  • if those shortcodes weren’t present in the raw input for that text node or attribute value, then escape the brackets

​@salvatorecorsi commented on ​PR #14770:


2 weeks ago
#40

That would give text and attributes the same rule: normalization must not introduce a shortcode that the raw input didn't contain. I'll prototype the comparison at those flush points, including encoded tag names and CSS-selector attributes. The registration-dependent parser is the part I still need to work through.

​@salvatorecorsi commented on ​PR #14770:


2 weeks ago
#41

I tried this in an isolated prototype. It catches encoded names in both text and attributes, and bracket escaping leaves the decoded CSS selector unchanged. I'd keep detection independent of registered tags so plugin activation doesn't change the saved result.

Counts still lose which occurrence was literal: ENCODED: &#91;probe&#93; LITERAL: [probe] can keep the first active and escape the second. Matching arguments doesn't resolve identical occurrences; preserving the literal one needs a link to its raw source.

There's also [pro<notallowed>be]: removing that disallowed tag creates [probe] across two text flushes. Should this PR cover those cases too, or specifically prevent new syntax within each individual text/attribute value?

This ticket was mentioned in ​PR #14841 on ​WordPress/wordpress-develop by ​@nofearinc.


2 weeks ago
#42

The HTML API based wp_kses() stops sanitizing at the first NOSCRIPT opening tag (break in sanitize()), so everything after it is dropped. For users without unfiltered_html (Authors, Contributors, commenters), a post or comment that contains a NOSCRIPT fallback loses the rest of its content on save:

Input 7.1.3 7.2-beta1
Before <noscript>[[Image(a.jpg)]]</noscript> after the noscript. Before [[Image(a.jpg)]] after the noscript. Before

This skips the NOSCRIPT element together with its contents and continues after its closer. Nothing inside a NOSCRIPT is emitted, so untrusted input still cannot produce one, including through a </noscript> hidden in an attribute value (one of the test cases). An unclosed NOSCRIPT still truncates, like other incomplete input.

Testing

  • vendor/bin/phpunit --filter test_wp_kses_removes_noscript_and_keeps_following_content: 5 of the 6 cases fail without the patch, all pass with it.
  • vendor/bin/phpunit --group kses (529 tests) and --group html-api (2051 tests) pass.
  • Merges cleanly with #14770.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Opus 5.5
Used for: finding the issue, drafting the fix and tests, and running them. I reviewed every line and reproduced the output above on 7.1.3 and 7.2-beta1.

---
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See ​GitHub Pull Requests for Code Review in the Core Handbook for more details.

🤖 Generated with ​Claude Code

#43 @nofearinc
2 weeks ago

In 7.2-beta1, wp_kses() stops sanitizing at the first NOSCRIPT opening tag (break in sanitize()), so for users without unfiltered_html everything after it is lost on save. Before <noscript><img src="a.jpg"></noscript> after the noscript. becomes Before , where 7.1.3 kept Before <img src="a.jpg"> after the noscript.

​https://github.com/WordPress/wordpress-develop/pull/14841 skips the NOSCRIPT element together with its contents and continues after its closer, so untrusted input still cannot produce one. An unclosed NOSCRIPT still truncates, in line with how incomplete input is handled since PR 13271. @dmsnell, was the break meant as a fail-closed choice? Happy to adjust.

AI assistance: Claude Code (Claude Opus 5.5) for finding the case and drafting the patch and tests; reviewed and tested by me.

#44 @dmsnell
13 days ago

@nofearinc the reason NOSCRIPT is rejected is because of the different parsing rules for scription-mode-enabled and scripting-mode-disabled parsing. there is one potential way to allow this, which I hadn’t considered before: upon finding a NOSCRIPT opening tag, find the scripting-mode-disabled NOSCRIPT closing tag and note it’s position. then, continue parsing until that position, and if the next token is anything other than the NOSCRIPT closing tag, reject to before it was opened.

the core issue is that in a script-mode-disabled environment like Core, there actually exist tags inside NOSCRIPT. in a script-mode-enabled environment like a browser with JS enabled, there are _no_ tags inside NOSCRIPT, meaning that the allowed HTML rules don’t apply, and parsing forward tag-by-tag could lead to unexpected breakout from the NOSCRIPT element.

<noscript><a title="</noscript>">oops</a></noscript>

in a script-mode-enabled environment there is no A element in the above snippet: the NOSCRIPT ends at the first </noscript>, and what follows is ">oops, an ignored unexpected </a>, and an ignored unexpected </noscript>.

in a script-mode-disabled environment there is an A element containing a title attribute whose value is </noscript>, whose contents are oops, and then nothing else.

so there’s no actual way to apply one set of rules against NOSCRIPT. given that browsers hide the NOSCRIPT contents with script-mode-enabled, it could make sense to clobber the contents, applying the script-mode-disabled parsing only. however, that relies on a private function skip_rawtext() which we weren’t sure we could expose properly for 7.2.

it is possible to match the NOSCRIPT closer in script-mode-enabled with a single PCRE pattern. ​using my PCRE-based HTML segmenter as an example.

<?php

if ( 'NOSCRIPT' === $token_name && ! $is_closer ) {
        $ws          = " \t\f\r\n";
        $avalue      = "(?:\"[^\"]*\"|'[^']*'|(?![\"'])[^>{$ws}]*)";
        $after_aname = "(?:[{$ws}]*+=[{$ws}]*+{$avalue}|(?![{$ws}]*+=))";
        $pattern     = "~</?noscript()(?:[{$ws}/]*(?:[^=/>{$ws}]+{$after_aname}|=[^=/>{$ws}]*{$after_aname}))*[{$ws}/]*>\K~i";

        if ( 1 === preg_match( $pattern, $this->html, $closer_match, PREG_OFFSET_CAPTURE, $here->start + $here->length ) {
                $noscript_end_at = $closer_match[0][1];
                $this->set_bookmark( 'inside_noscript' );
                while ( $this->next_token() ) {
                        if ( $this->get_span()->start < $noscript_end_at ) {
                                continue;
                        }

                        if ( 
                                $this->get_span()->start > $noscript_end_at ||
                                'NOSCRIPT' !== $this->get_tag() || 
                                ! $this->is_tag_closer()
                        ) {
                                /*
                                 * Only allow NOSCRIPT if the script-mode-enabled
                                 * and script-mode-disabled boundaries align.
                                 */
                                break 2;
                        }


                        break;
                }
                $this->seek( 'inside_noscript' );
        }
}

This is crude, and the code is there for concept only. In a real implementation we would want to avoid the seek and instead track state as being inside NOSCRIPT, and storing $before_noscript, processing once, and then truncating if the two don’t align (see SVG/MathML handling for an example).

It would also need to remove the entire NOSCRIPT contents if NOSCRIPT weren’t allowed.

Feel free to work on this. I hope this conveys the rationale for not entering into it. We can always make wp_kses() more robust, but in this rewrite, we want to be cautious and certain before we allow HTML patterns we can’t fully verify.

​@dmsnell commented on ​PR #14770:


13 days ago
#45

I'd keep detection independent of registered tags so plugin activation doesn't change the saved result.

this is an easy escape hatch, and something I plan on proposing, but it’s a backwards-compatibility break with unknown consequences. shortcode detection is kind of a wild mess.

removing that disallowed tag creates [probe] across two text flushes

you are starting to see the complicated cases this touches. another change to shortcodes would free this up, and I think the _intent_ is that shortcodes today only exist within complete text nodes, or inside HTML attributes, but the current shortcode parser doesn’t do that.

---

and this is all work meant to ensure the prevention of creating shortcodes, yet we have to deal with representing non-shortcode lookalikes.

a good metric might simply be that we create nothing that could parse as a shortcode through wp_kses() that wasn’t already there. it’s not right to escape shortcodes by using the escaped brackets (because this is how CSS selectors, for example, need to be represented); just as it’s not right to escape everything that looks like a shortcode using double brackets (because things like CSS selectors are not shortcodes, so shouldn’t be turned into them).

this all brings me back to shortcodes, where the heart of the problem is. we’re supposed to escape anything that could appear like a shortcode, meaning that if we ever expect data-selector="[hidden]", we should be writing it like data-selector="[[hidden]]", except then we have to rely on shortcode processing that content before it gets sent to a browser, or to an API response, which often doesn’t happen.

​@salvatorecorsi commented on ​PR #14770:


12 days ago
#46

You're right that registration-independent detection would change compatibility, so I won't build that assumption into this patch. The prototype only checked the decoded selector value; it didn't establish that every shortcode-processing path remains compatible. I'll keep the implementation unchanged for now, until there's a rule that prevents newly created shortcode syntax without rewriting literal selector values, including when removing markup joins text across tokens.

Note: See TracTickets for help on using tickets.