Shortcode Support Regained but Content Filters are messing with Shortcode HTML

[domainsupport]

I am extremely grateful that the Security Team were able to quickly regain support for shortcodes in the Block Theme templates. However, whatever change has been agreed and pushed out means that the filters to automatically inject <br> and <p> tags into the content are now affecting shortcodes and we are seeing text being automatically wrapped in <p> tags and carriage returns replaced with <br> tags.

Rather than revert to the insecure v6.2.1 we are going through shortcodes to remove any carriage returns.

Please advise.

Oliver

[kubiq]

Here you can see all possible usecases and current problems:
​https://demo.kubiq.sk/sample-page/

This demo uses [xxx] shortcode that should render A B C

<?php
add_shortcode( 'xxx', function(){
        return "A\nB\nC";
});

1. Shortcode block in Template and Template part is doing redundant autop
2. Shortcodes are not working when they are in reusable block in Template and Template part (group, row, whatever)

This is clean WP install without any plugin and default 2023 theme - I've just added functions.php in it, with the code from above

[Clorith]

I'm not sure if this scenario would be considered a bug, or that the previous behavior would be considered an unintentional lack of processing.

The \n symbol represents a newline after all, but wasn't being treated as such inline since processing has already happened to the post content at the time of the shortcode being called.

Given that the site editor is a whole other beast, that shortcodes do not work exactly the same as in post content feels like something we can communicate and make clear that they are two very different use cases 🤔

As for the reusable block scenario, that does appear unintentional, and those may need to also intentionally call do_shortcode and make it an explicit expectation that shortcodes within them are rendered (this is the recommendation for other blocks as well, so seems like a sensible thing to introduce I think).

[kubiq]

@Clorith shortcodes should never do any autop! Imagine pretty common output for input with a lot of attributes

<input
    type="number"
    name="age"
    value=""
    placeholder="18"
    id="age"
    class="form-control"
    min="18"
    max="80"
    step="1"
/>

any many other scenarios... \n inside the shortcodes has to stay \n

[domainsupport]

@Clorith I think you'll find that @kubiq is correct on this one I'm afraid; and I have a better example ...

<textarea>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</textarea>

... would have been perfectly fine FSE template shortcode output in v6.2 but in 6.2.2 it is now filtered and is output something like this ...

<textarea>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

<p>Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.</p>

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
<br>
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</textarea>

... and there's nothing that can be done about it.

I'm not sure why there haven't been more complaints about this yet but I am sure that there will be.

Oliver

[aflijja]

I second @domainsupport; here is another one ...

<p>        <script>
        jQuery(document).ready(function($) {</p>
<p>            $('#wcps-558').slick({
                adaptiveHeight: true,
                slidesToShow: 4,
                slidesToScroll: 4,
                responsive: [{
                        breakpoint: 1200,
                        settings: {
                            slidesToShow: 4,
                            slidesToScroll: 4,
                            nav: true,
                        }
                    },
                    {
                        breakpoint: 768,
                        settings: {
                            slidesToShow: 2,
                            slidesToScroll: 2,
                            nav: true,
                        }
                    },
                    {
                        breakpoint: 576,
                        settings: {
                            slidesToShow: 1,
                            slidesToScroll: 1,
                            nav: true,
                            touchMove: true,
                            swipeToSlide: true,</p>
<p>                        }
                    }
                ],
                gutter: 10,
                arrows: true,
                appendArrows: '.controlsWrap-558',
                prevArrow: '</p>
<div class="prev"><i class="fas fa-chevron-left"></i></div>
<p>',

the strange part for me is this problem exist only on the front page.

[jim5471]

I have a number of shortcode blocks in templates and template parts and am not seeing problems with the latest release.

Are you putting your custom HTML inside Custom HTML Blocks ?

I don't use reusable blocks so this may be a stupid question.

What is wrong with using "convert to blocks" ?

Then the shortcodes start working.

[domainsupport]

Replying to jim5471:

I have a number of shortcode blocks in templates and template parts and am not seeing problems with the latest release.

Are you putting your custom HTML inside Custom HTML Blocks ?

I don't use reusable blocks so this may be a stupid question.

What is wrong with using "convert to blocks" ?

Then the shortcodes start working.

HTML generated by our shortcodes is having <p> and <br> tags injected into it after it's been generated as if it were post content. This is not static HTML that can be moved into a Custom HTML block otherwise we wouldn't be using a shortcode at all. The HTML is dynamic and requires PHP to generate it.

Oliver

[aflijja]

a temp solution for me
patch "wp-includes/block-template.php"
with 2 lines

	$content = str_replace( ']]>', ']]>', $content );
	$content = str_replace( '<p>', '', $content );
	$content = str_replace( '</p>', '', $content );

the second problem is new line generation inside script tag
i changed the single quote ' to backtik `
so accidental multiline script work

    prevArrow: `
<div class="prev"><i class="fas fa-chevron-left"></i></div>
`,

but this is temporary solution for me

[bekk]

This is also causing <p> tags to appear within inline javascript output by some shortcodes

[jim5471]

@domainsupport, @bekk - Just trying to understand the issue..

If you return inline script in this way do you get problems ?

<?php
add_shortcode( 'xxx', function(){
        return "<!-- wp:html --><script><!-- code here /--></script><!-- /wp:html -->";
        });

[domainsupport]

Replying to jim5471:

If you return inline script in this way do you get problems ?

<?php
add_shortcode( 'xxx', function(){
        return "<!-- wp:html --><script><!-- code here /--></script><!-- /wp:html -->";
        });

I didn't wait to see if inline script was affected although I have no doubt that it is. I started trying to overcome the issue by putting all returned HTML on a single line but then realised that this wasn't possible for <textarea> input fields that have paragraphs of text within them and at that point I gave up.

I notice you are getting your example xxx shortcode to output an wp:html formatted block ... are you saying that this would be ignored by the content filters and could be a possible work-around?

If this is what you meant in your original comment then I apologise as I didn't realise.

[bekk]

@jim5471 yes it gets rendered as

<p><script><!-- code here /--></script></p>

Likewise if you change it to:

<?php
add_shortcode( 'xxx', function() {
    return "<!-- wp:html --><script>
var obj = {
    'foo': 'bar',
    'baz': 'qux'
};

var json = JSON.stringify(obj);
</script>
    <!-- /wp:html -->";
});

It's rendered as:

<p><script>
var obj = {
    'foo': 'bar',
    'baz': 'qux'
};</p>
<p>var json = JSON.stringify(obj);
</script></p>

[jim5471]

@domainsupport

OK - now try the code without any line feeds.
It is being wrapped in <p> tags but the code is clean.

<?php
add_shortcode( 'xxx', function() { 
        return "<!-- wp:html --><script>var obj = { 'foo': 'bar', 'baz': 'qux'};var json = JSON.stringify(obj);</script><!-- /wp:html -->";
});

<p><script>var obj = { 'foo': 'bar', 'baz': 'qux'};var json = JSON.stringify(obj);</script></p>

[bekk]

Replying to jim5471:

@domainsupport

OK - now try the code without any line feeds.
It is being wrapped in <p> tags but the code is clean.

<?php
add_shortcode( 'xxx', function() { 
        return "<!-- wp:html --><script>var obj = { 'foo': 'bar', 'baz': 'qux'};var json = JSON.stringify(obj);</script><!-- /wp:html -->";
});

<p><script>var obj = { 'foo': 'bar', 'baz': 'qux'};var json = JSON.stringify(obj);</script></p>

Right, see ​https://developer.wordpress.org/reference/functions/wpautop/

[jim5471]

OK - just found one difference between the example we have been looking at and the way I have coded my shortcodes.

<?php
function do_xxx() {
       $html = "<!-- wp:html --><script>
           var obj = { 'foo': 'bar', 'baz': 'qux'};
           var abc = 'xcxcfvgbcxbvcxbvxc';
           var json = JSON.stringify(obj);
           </script><!-- /wp:html -->";
        return $html;
    }
    add_shortcode('xxx', 'do_xxx');

Returns this...

<p><script>
           var obj = { 'foo': 'bar', 'baz': 'qux'};
           var abc = 'xcxcfvgbcxbvcxbvxc';
           var json = JSON.stringify(obj);
           </script></p> 

[domainsupport]

Replying to jim5471:

OK - just found one difference between the example we have been looking at and the way I have coded my shortcodes.

<?php
function do_xxx() {
       $html = "<!-- wp:html --><textarea>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</textarea><!-- /wp:html -->";
        return $html;
    }
    add_shortcode('xxx', 'do_xxx');

Returns this...

<p><textarea>Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.</p>
<p>Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.</p>
<p>Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.</p>
Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</textarea></p>

[davidsjptimoney]

Another use case. Our site is for a non-profit film club. We use Eventbrite for ticketing, specifically their Embedded Checkout modal, which looks like this:

<!-- Noscript content for added SEO -->
<noscript><a href="https://www.eventbrite.co.uk/e/aftersun-12-tickets-590034829057" rel="noopener noreferrer" target="_blank">Buy Tickets on Eventbrite</a></noscript>
<!-- You can customise this button any way you like -->
<button id="eventbrite-widget-modal-trigger-590034829057" type="button">Buy Tickets</button>

<script src="https://www.eventbrite.co.uk/static/widgets/eb_widgets.js"></script>

<script type="text/javascript">
    var exampleCallback = function() {
        console.log('Order complete!');
    };

    window.EBWidgets.createWidget({
        widgetType: 'checkout',
        eventId: '590034829057',
        modal: true,
        modalTriggerElementId: 'eventbrite-widget-modal-trigger-590034829057',
        onOrderComplete: exampleCallback
    });
</script>

This is held in an ACF field for the film (a post). After the 6.2.2 fix, this is parsed and the line-feeds & carriage returns replaced with <br> and <p>.

To work around this, we've had to add extra code to the PHP snippet that pulls the ACF field and customises the button. Thus:

$searchf = array("\n","\r");
$content = str_replace($searchf,"",$content);

The other thing we noticed was that a table cell that wasn't properly closed (i.e. <tr><td></tr> instead of <tr><td></td></tr>) got a </p> injected, which caused layout issues.

We use a customised twenty twenty-two theme (not a child theme). Some of our site fell over with 6.2.1 but we got round that by encapsulating the shortcodes in the template within template parts, though I'm not clear if that is a sutainable solution, to judge from some of the other comments I've seen on here.

[desrosj]

Marking has-testing-info, but please do provide as many detail as you can. If you've reported above and left out important details, feel free to clarify and expand.

Also marking needs-unit-tests. Ideally we start writing more tests for these edge cases to help us ensure they're all resolved appropriately. Probably a good situation for test first development.

[jim5471]

@domainsupport

OK now I get it !

[ryansantschi]

Having this issue as well. Not sure if this goes without saying, but any HTML comments (<!--) also are put in paragraph tags.

In my particular use case this is providing a temporary work around:

// Replace one or more consecutive whitespace characters with a single space
$shortcode_content = preg_replace('/\s+/', ' ', $shortcode_content);
// Remove HTML comments from the content
$shortcode_content = preg_replace('/<!--(.|\s)*?-->/', '', $shortcode_content);

[tdrayson]

I have been doing some testing, it seems that it's only the shortcode "block" that has this formatting issue.

If you put your shortcode into a template with a paragraph block, everything gets rendered correctly.

Backend - ​https://share.getcloudapp.com/kpul9w0K
Frontend - ​https://share.getcloudapp.com/YEu4qyLL

Not a perfect solution as your shortcode will get wrapped with a <p> tag.

[kubiq]

@tdrayson shortcode block in Template and Template part,
but also take a look on my demo:
​https://demo.kubiq.sk/sample-page/
and comment here
https://core.trac.wordpress.org/ticket/58366#comment:3
Shortcodes in reusable blocks are completely broken, no matter how you add them, once they are in Template or Template part

[Ov3rfly]

Another use case similar to @davidsjptimoney, javascript snippets added via ACF field.

We have not (yet) the problem described here, but obviously only due to (still) use of classic theme instead of (planned) block theme.

We can not simply strip \n and \r from code as our javascript snippets also contain lines with // javascript comments similar to example here: ​https://stackoverflow.com/q/65274912

<script>
// Set to the same value as the web property used on the site
var gaProperty = 'UA-XXXX-Y';

// Disable tracking if the opt-out cookie exists.
var disableStr = 'ga-disable-' + gaProperty;
if (document.cookie.indexOf(disableStr + '=true') > -1) {
  window[disableStr] = true;
}

// Opt-out function
function gaOptout() {
  document.cookie = disableStr + '=true; expires=Thu, 31 Dec 2099 23:59:59 UTC; path=/';
  window[disableStr] = true;
}
</script>

These kind of code snippets with // javascript comments lines are widely in use and need their linefeeds present.

[bekk]

I haven't been able to come up with a simple enough patch that doesn't cause side effects. For anyone needing a temporary workaround mine is below it excludes the wpautop'ing for only the shortcode block, also @tdrayson's works too with the caveat that the shortcode is <p>'d

<?php
add_filter('register_block_type_args', function ($settings, $name) {
        if ($name === 'core/shortcode') {
                $settings['render_callback'] = function ($attributes, $content) {
                        return $content;
                };
        }
        return $settings;
}, 10, 2);

[kubiq]

Replying to bekk:

Nice, that will fix that <p>s and <br>s... Thanks for the effort!

But there is still one big issue - shortcodes in reusable blocks in Templates and Templat parts are not rendering at all... Imagine some CTA section with Newsletter form from Contact Form 7 or anything like that = big problem on many many websites

[andreasjr]

I'm finding that 6.2.2 broke our shortcodes even more so than normal (we were at least able to fix rendering them by sticking them in Template Parts). As @kubiq mentioned, we use a lot of Newsletter signup forms in our CTA/Footer sections, and Gravity Forms' "block" is just a wrapper around a shortcode. This means all our footers just show a shortcode snippet instead of the form.

[andreasjr]

Here is a more detailed finding of the problem me and my team are facing.

Some context: We are specifically experiencing problems with the Gravity Forms block. The GF block looks like it’s a wrapper for a shortcode. Obviously this is not WP team’s responsibility to fix, and ultimately GF will need to fix this, but it should be noted this block worked fine before the 6.2.x updates.

In a page:

• GF block works
• Shortcode works

Reusable block on a page:

• GF block works
• Shortcode works

In a template (e.g. page.html):

• GF block does not work
• Shortcode works

Reusable block in a template (e.g. page.html):

• GF block does not work
• Shortcode does not work

Template part in a template (e.g. part.html in page.html):

• GF block does not work
• Shortcode works

Reusable block in a template part in a template (e.g. part.html in page.html):

• GF block does not work
• Shortcode does not work

Additionally, when the shortcode does work, as others have mentioned there are extra <br> and <p> tags added.

[aflijja]

Replying to bekk:
thank you it work.

[coreyw]

Also experiencing this with 6.2.2 and the ​WP Notification Bell plugin.

​https://pastebin.com/bZjP5VAE

Notice errant paragraph tags inside script, surrounding spans, etc.

[peterwilsoncc]

POC -- not for commit

[peterwilsoncc]

In [33469] for #33106, a placeholder comment was added to wpautop() to protect new lines within CDATA.

In 58366-poc.diff​ I've done something similar for protecting new lines within shortcodes for block themes. It uses the slightly more verbose wp-block-theme-line-break-placeholder to avoid collisions with wpautop placeholders.

I've done some minimal testing using the shortcode in comment 3 but if the approach is validated, it will need a rewrite as the POC uses closures unnecessarily.

[christopherplus]

Also experience this in 6.2.2 with a shortcode that outputs WP_Query.

[bekk]

For review

[bekk]

My patch draws on the observation that the shortcode block should not be autop'ing its own content. It is noted in the_content filters that shortcodes should be processed after wpautop(), I don't see why the shortcode block is any different.

I also added shortcode processing to the core block, which restores shortcodes in template parts and reusable blocks.

My patch is attached above for review

[stiofansisland]

We are also having issues. Nested shortcodes no longer act as they did before, now nested shortcodes show the ending trailing shortcode. All issue seem to stem from this change: ​https://github.com/WordPress/WordPress/commit/2bb3a5169548d16173cf48ca9da1111efc428f86

The shortcode calls were put back in the wrong place, everything will be fixed for everyone if this is restored to how it was in 6.2.0. Unless there is a security reason not to do this, please do it?

[kaylam]

Replying to andreasjr:

Here is a more detailed finding of the problem me and my team are facing.

Some context: We are specifically experiencing problems with the Gravity Forms block. The GF block looks like it’s a wrapper for a shortcode. Obviously this is not WP team’s responsibility to fix, and ultimately GF will need to fix this, but it should be noted this block worked fine before the 6.2.x updates.

I am also having ongoing issues with the Gravity Forms block from the recent update. At first I could get the form back by putting it in a Template Part. Now, as @andreasjr outlined, in a Template or Template Part, I can get the form to show up by using a core Shortcode Block instead, but there's <p> and <br> in the resulting html that should not be there.

[ryno267]

had to rollback.
I'm hoping this hits /milestone/6.2.3

[huubl]

Replying to bekk:

My patch draws on the observation that the shortcode block should not be autop'ing its own content. It is noted in the_content filters that shortcodes should be processed after wpautop(), I don't see why the shortcode block is any different.

I also added shortcode processing to the core block, which restores shortcodes in template parts and reusable blocks.

My patch is attached above for review

This seems to to working (when Gutenberg plugin is not active).

[askcompgeek]

I had to rollback to 6.2. I hope this is fixed soon!

[brumack]

Replying to bekk:

My patch draws on the observation that the shortcode block should not be autop'ing its own content. It is noted in the_content filters that shortcodes should be processed after wpautop(), I don't see why the shortcode block is any different.

I also added shortcode processing to the core block, which restores shortcodes in template parts and reusable blocks.

My patch is attached above for review

This unfortunately doesn't address blocks using shortcodes saved to block attributes and rendered as html *after* do_blocks is called, so anything using serverside rendering and used inside a template part will not benefit.

When used inside post_content, these serverside rendered shortcodes are processed normally, so there's a discrepancy between how blocks are rendered in post content and in template parts.

[rilwis]

We experienced a ​similar problem with shortcode and wpautop recently with our ​MB Views plugin.

After investigating, we found that in a block theme, there are 2 issues:

1. The "shortcode" block uses wpautop to ​wrap content in paragraphs. I have no idea why is that. In my opinion, this should be removed.
2. The process order of rendering block templates changed and caused the bug. See these 2 commits:

​https://github.com/WordPress/WordPress/commit/6a077b35f15590a843ff8a6c97a135f3a34872dd
​https://github.com/WordPress/WordPress/commit/2bb3a5169548d16173cf48ca9da1111efc428f86

Previously, the content of a block template goes through do_blocks ("shortcode" blocks apply wpautop, but since shortcodes are not parsed, nothing happens), then shortcode_unautop, then do_shortcode, which doesn't cause the bug.

However, after the change, the content goes through shortcode_unautop, do_shortcode (shorcodes are pased here), then do_blocks (and "shortcode" blocks apply wpautop here).

My suggestions:

1. Remove applying wpautop for the shortcode block
2. Correct the order of processing content for block templates

[askthecompgeek]

After upgrading my website to WordPress 6.2.2, I followed the suggestion of placing my shortcode in a template part and it worked for one website but not another. I'm really new to WordPress so I didn't know how to setup a debug environment in order to debug the code so I went old school brute force and started comparing components of each website. Through trial and error I discovered I had to do two things in order for a shortcode to work in websites running 6.2.2:

1. Place the shortcode in a template part and use that template part where you would use the shortcode
2. Include the plugin Gutenverse

I can't tell you why adding Gutenverse solved the issue but hopefully the experts can quickly discover what this plugin did and release a 6.2.3 patch so we don't have to run this plugin if we don't need it.

https://core.trac.wordpress.org/ticket/58366

[askthecompgeek]

Replying to tdrayson:

I have been doing some testing, it seems that it's only the shortcode "block" that has this formatting issue.

If you put your shortcode into a template with a paragraph block, everything gets rendered correctly.

Backend - ​https://share.getcloudapp.com/kpul9w0K
Frontend - ​https://share.getcloudapp.com/YEu4qyLL

Not a perfect solution as your shortcode will get wrapped with a <p> tag.

Thanks... The extra formatting messes things up a bit but it does work. I also noticed that if you enable the Gutenverse plugin, it works with no code changes. I have no idea why--I'm still pretty new to WordPress.

[askthecompgeek]

Replying to bekk:

I haven't been able to come up with a simple enough patch that doesn't cause side effects. For anyone needing a temporary workaround mine is below it excludes the wpautop'ing for only the shortcode block, also @tdrayson's works too with the caveat that the shortcode is <p>'d

<?php
add_filter('register_block_type_args', function ($settings, $name) {
        if ($name === 'core/shortcode') {
                $settings['render_callback'] = function ($attributes, $content) {
                        return $content;
                };
        }
        return $settings;
}, 10, 2);

That fixed my short code issue in WordPress 6.2.2. Thank you!

[ryno267]

Running 6.3-RC1, no change, same issue with many <br>'s added.

Setup is a menu of pages via code function:
I call that via a shortcode block that's in a sidebar/aside template part.
In 6.2.1 and 6.2.2 that stopped working and only the shortcode [name_here] text displays.
In 6.3-RC1 that now displays the menu but with all the BR's added.

Doing the "hack" where you make the shortcode block in a template part... and adding that template-part to the sidebar template-part (instead of the shortcode directly)...
This made it work in 6.2.1, 6.2.2, and 6.3-RC1, but all the BRs are added again.

...I see this isn't in 6.3 so that's concerning; but there's still an issue to resolve here.

[domainsupport]

Testing with v6.3-RC1 as @ryno267 mentioned the issue has not been resolved.

I've opted to go with @bekk's add_filter() solution above (which I've added to our "Options for Block Themes" plugin) and so far it looks like it's working which is great but it would be good to see this fixed in core soon.

[ryno267]

Replying to domainsupport:

I've opted to go with @bekk's add_filter() solution above

Yeah that does work for now, thx for that callout – need to test this hard today, but ugh... is what it is for now. This code comment should be fun...

[audrasjb]

Adding to milestone 6.4 to hopefully help this ticket to get a patch ready in time for the next major release.

[kubiq]

Please notice, that there are more problems, not just line breaks.
I made my demo page more understandable (hopefully), so here is the link again:
​https://demo.kubiq.sk/sample-page/

[kubiq]

and there is another weird problem...
if you have a WP_Query loop inside the shortcode and you try to call the_content() then it seems that 'the_content' filter is not triggered for the first item, but it works for all other items then...
I can fix this by calling apply_filters( 'the_content', '' ); before the loop

[ryno267]

still showing as 'new', not accepted... 👀
https://core.trac.wordpress.org/query?component=Shortcodes&milestone=6.4&group=status&col=id&col=summary&col=owner&col=type&col=priority&col=component&col=version&order=priority

[nicolefurlan]

This ticket was discussed during the 6.4 bug scrub today. The submitted POC patch needs review from others so that we can figure out how best to solve the root cause of the problem. I'm hoping that this comment reopens this discussion so that we can potentially get a fix in for 6.4.

@hellofromTonya mentioned that the next patch needs a lot of thought and care to avoid causing more problems.

[nicolefurlan]

I don't think this ticket is going to make it into this release cycle since RC1 is only one week away. If there are no objections, I will move this to 6.5.

[ryno267]

@nicolefurlan I object! ;) It feels like an important issue with a working hack but I understand dev cycles and making releases. I just really hope it doesn't miss 6.5...

[peterwilsoncc]

Replying to ryno267:

@nicolefurlan I object! ;) It feels like an important issue with a working hack but I understand dev cycles and making releases. I just really hope it doesn't miss 6.5...

At the moment 58366-poc.diff​ is available as a proof of concept but needs further testing to validate it fixes the issue with line breaks.

If it proves successful, I'll work on a pull request to get the code in a form that is ready for commit (the POC is really, really hacky).

As getting this right has proven difficult, I agree with @nicolefurlan that getting this in to the 6.4 cycle is best avoided with the release candidate due next week.

It would be helpful to get some testing of the POC, these are some of the things that need to be tested:

• short codes are not executed in user submitted content (comments and other form data)
• that it resolves the issue with line-breaks being stripped from shortcodes in block themes and replaced with HTML tags
• paragraph and line break tags are not added inappropriately around the shortcode
• content is not stripped for users without the unfiltered_html capability

I really would like to get this fix in but getting the POC validated, converting it to a suitable patch and writing up unit tests within the week is not possible.

If a few folks could test 58366-poc.diff​ and see if it solves the problems without reintroducing the security issues that would be most helpful.

[giorgosm]

Can i ask which is the working hack for this? Thank you

Replying to ryno267:

@nicolefurlan I object! ;) It feels like an important issue with a working hack but I understand dev cycles and making releases. I just really hope it doesn't miss 6.5...

[domainsupport]

Replying to giorgosm:

Can i ask which is the working hack for this? Thank you

We use the below fix in our Options for Block Themes plugin ...

Replying to bekk:

I haven't been able to come up with a simple enough patch that doesn't cause side effects. For anyone needing a temporary workaround mine is below it excludes the wpautop'ing for only the shortcode block, also @tdrayson's works too with the caveat that the shortcode is <p>'d

<?php
add_filter('register_block_type_args', function ($settings, $name) {
        if ($name === 'core/shortcode') {
                $settings['render_callback'] = function ($attributes, $content) {
                        return $content;
                };
        }
        return $settings;
}, 10, 2);

[samuel1337]

Replying to peterwilsoncc:

Based on the fixes. Why don't we put the break lines within the HTML tags? As a User, I still want the HTML tags if newline exists within the HTML tags. Especially within the Gutenberg Shortcode Blocks.

I put this issue a couple of days ago: ​https://github.com/WordPress/gutenberg/issues/56617

Replying to ryno267:

@nicolefurlan I object! ;) It feels like an important issue with a working hack but I understand dev cycles and making releases. I just really hope it doesn't miss 6.5...

At the moment 58366-poc.diff​ is available as a proof of concept but needs further testing to validate it fixes the issue with line breaks.

If it proves successful, I'll work on a pull request to get the code in a form that is ready for commit (the POC is really, really hacky).

As getting this right has proven difficult, I agree with @nicolefurlan that getting this in to the 6.4 cycle is best avoided with the release candidate due next week.

It would be helpful to get some testing of the POC, these are some of the things that need to be tested:

• short codes are not executed in user submitted content (comments and other form data)
• that it resolves the issue with line-breaks being stripped from shortcodes in block themes and replaced with HTML tags
• paragraph and line break tags are not added inappropriately around the shortcode
• content is not stripped for users without the unfiltered_html capability

I really would like to get this fix in but getting the POC validated, converting it to a suitable patch and writing up unit tests within the week is not possible.

If a few folks could test 58366-poc.diff​ and see if it solves the problems without reintroducing the security issues that would be most helpful.

[peterwilsoncc]

@samuel1337 That's what 58366-poc.diff​ attempts to do but it needs further testing before being committed.

[gregfuller]

I just want to put in my 2 cents worth, as I think it's a very serious problem. We don't have blocks for everything and must rely on shortcodes. A number of shortcodes are not working for me, including the Rankmath Breadcrumb shortcode.

[rajinsharwar]

Unfortunately, we do not have much activity on this to be able to land in 6.5. So, I am moving this to 6.6 to get some early eyes.

[mirkolofio]

Replying to gregfuller:

[...] I think it's a very serious problem.

Indeed. It's a serious one.

Replying to rajinsharwar:

Unfortunately, we do not have much activity on this to be able to land in 6.5. So, I am moving this to 6.6 to get some early eyes.

Sad, but hopefully we can see a nice working patch and a fix in 6.5.

Meanwhile, there's a patch provided by WooCommerce.
​https://github.com/woocommerce/woocommerce/blob/145d75a08fc607178ed4bda1f482b4964c4adecb/plugins/woocommerce/src/Blocks/BlockTemplatesController.php#L52-L78

Just change line 64 accordingly.

[bekk]

Replying to mirkolofio:

Meanwhile, there's a patch provided by WooCommerce.
​https://github.com/woocommerce/woocommerce/blob/145d75a08fc607178ed4bda1f482b4964c4adecb/plugins/woocommerce/src/Blocks/BlockTemplatesController.php#L52-L78

Just change line 64 accordingly.

This does the same as my workaround posted near the top of this thread except it's scoped to woocommerce blocks.

Hopefully these fundamental and technical issues with blocks get resolved at some point. We're nearing a 1 year bug anniversary on this one

Committed in r57943 / ​https://github.com/WordPress/wordpress-develop/commit/18db904c293f43f5e928b74835d0969cd60c1e2e

[oglekler]

@kubiq This is a commit to this ticket. Can you, please, check that this is solving the issues you had? Thank you!

[kubiq]

Hi @oglekler, you mean these two files?
​https://github.com/WordPress/wordpress-develop/commit/18db904c293f43f5e928b74835d0969cd60c1e2e
I've downloaded them and replaced in /wp-admin/ folder, but I don't see any difference:
​https://demo.kubiq.sk/sample-page/

[oglekler]

Sorry @kubiq it looks like this commit is attached to this ticket accidentally and has nothing to do with shortcodes 😳

Great that you have a sample page 👍 and it looks like I am not seen it the first time 🤦

Sadly, it looks like we will not be able to fix this in this milestone, so, I am moving it to the next one. 😐

[kubiq]

Replying to peterwilsoncc:

At the moment 58366-poc.diff​ is available as a proof of concept but needs further testing to validate it fixes the issue with line breaks.

It will fix the line breaks but there are still completely not working shortcodes in Reusable blocks in Templates or Template parts: ​https://demo.kubiq.sk/sample-page/

It's broken for 1.5 years! We really need to fix this somehow...

[kubiq]

So the problem is that reusable blocks (patterns) used in templates and template part and containing shortcodes will never run do_shortcode function.

My quickfix solution is to add do_shortcode into the do_blocks function in \wp-includes\blocks.php so eg. instead of return $output; we would have return do_shortcode( $output );

Or maybe it can be done in render_block function if we need some conditional triggering of that do_shortcode... I'm not sure how "resources expensive" is that function, I didn't measure it.

[kubiq]

And here is my hotfix solution for live websites where I don't want to change WP core files:

<?php
add_filter( 'render_block', function( $block_content, $parsed_block, $block ){
        return do_shortcode( $block_content );
}, 10, 3 );

[mikinc860]

Replying to kubiq:

And here is my hotfix solution for live websites where I don't want to change WP core files:

<?php
add_filter( 'render_block', function( $block_content, $parsed_block, $block ){
        return do_shortcode( $block_content );
}, 10, 3 );

<?php
add_filter( 'render_block', function( $block_content, $parsed_block ) {
    return do_shortcode( $block_content );
}, 10, 2 );

I removed the third parameter, $block, because it's not needed based on the current usage. The render_block filter provides only two arguments by default.

[peterwilsoncc]

I've moved this off the milestone as the POC needs further review.