shortcode block in block-based template part in a classic theme does not get expanded

[pbiron]

While testing the new feature in 6.1 (w/ 6.1-beta3) for a classic theme to include block-based template parts, I discovered that shortcode blocks in such template parts are not expanded (their render callbacks are not called).

The details are in my ​comment on the Dev Note announcing this new feature.

Note: creating the ticket here in Trac, rather than the GB GitHub, because that's where @mamaduka asked me to open in

[manfcarlo]

See also: #54614, #54759, #55996

[desrosj]

Moving to 6.1 milestone for investigation.

[chaion07]

@pbiron thank you for reporting this. We reviewed this ticket during a recent bug-scrub session and based on the feedback we are adding the keyword needs-testing.

Cheers!

Props to @costdev & @mukesh27

[pbiron]

minimal classic theme that uses block-based template part with a shortcode block

[pbiron]

test-56780.zip​ contains a minimal classic theme that can be used to replicate the problem.

1. Download and activate the theme on a site running WP 6.1-beta1, 6.1-beta2, 6.1-beta3, or 6.1-RC1.
2. Then view any post/page on the front-end.
3. Expected results: the string 'hellow world' should display above any post/page content.
4. Actual results: the stirng '[hello_world]' displays above any post/page content.

[audrasjb]

As per today's bug scrub, we're going to move it to milestone 6.2 with high priority and early workflow keyword.

This is a very annoying bug, but if we try to fix it during RC cycle, we're probably going through a rabbit hole (see #54759, #55996).

[ironprogrammer]

Reproduction Report

I was able to reproduce this issue.

Environment

• Hardware: MacBook Pro Apple M1 Pro
• OS: macOS 12.6.1
• Browser: Safari 16.1
• Server: nginx/1.23.2
• PHP: 7.4.33
• WordPress: 6.2-alpha-54642-src
• Theme: test-56780 v0.1.0 (the provided attachment:test-56780.zip​)
• Active Plugins:
  • gutenberg v14.8.0-rc.1 (tested active/inactive)

Actual Results

• ✅ Shortcodes are not processed when included in a block_template_part.

Additional Information

• Shortcode was not processed in the template part, whether inside a <!-- wp:shortcode --> or standalone.
• Added paragraph and shortcode blocks to post to demonstrate the shortcode was otherwise functional.
• Applied the following adjustments to the provided test theme for added clarity:
  • in masthead.html, added <h3>masthead:</h3> before the shortcode block
  • correct typo in shortcode to [hello_world]
  • in functions.php, updated return value of shortcode to hello world!

Supplemental Artifacts

[audrasjb]

As per today's bug scrub and since this ticket doesn't have any patch yet, let's move it to Future Release.

Update block_template_part to expand shortcodes in template parts

Add set of typical actions run on the_content to the block_template_part function. This replicate the behavior of the template part block's render method.

Trac ticket: https://core.trac.wordpress.org/ticket/56780

[petitphp]

Added a patch that replicate the behavior of the template part block's render method in block_template_part. Tested manually and with the newly created unit test.

@audrasjb I don't know if this ticket can be added back to the 6.2 milestone or if it's too late given it was tagged early.

[ironprogrammer]

Thanks for the patch, @petitphp! I've expanded the test cases to include the other processing ​added in PR 3851.

Test Report

Patch tested: ​https://github.com/WordPress/wordpress-develop/pull/3851

Environment

• Hardware: MacBook Pro Apple M1 Pro
• OS: macOS 12.6.2
• Browser: Safari 16.2
• Server: nginx/1.23.3
• PHP: 7.4.33
• WordPress: 6.2-alpha-54642-src
• Theme: test-56780 v0.1.0 (same as comment:14 plus changes noted below)

Actual Results

block_template_part content is now processed by:

• ✅ do_shortcode (original reported issue)
• ✅ wptexturize (e.g. processed smart quotes)
• ✅ convert_smilies
• ✅ shortcode_unautop
• ✅ ​wp_filter_content_tags (added decoding="async" loading="lazy" attributes to <img> tag)
• ✅ $wp_embed->autoembed

Additional Notes

• In the test theme, I replaced masthead.html with ​the content of this gist to incorporate the additional markup processing.

Supplemental Artifacts

Before Patch	After Patch

Thanks for this, @petitphp! Because this PR goes beyond handling do_shortcode() (original issue), it would be great if the unit tests accounted for the other changes. If you have time to expand those further, perhaps we can nudge this back into the current release cycle 🤞🏻

There are still ​a couple of `early` scrubs left, so I don't think the door is closed quite yet if this keeps up momentum.

@ironprogrammer Thanks for taking the time to do another round of testing! As suggested, I've expanded the test cases for the change based on the ones you mentioned on your report.

[hellofromTonya]

I don't think this ticket needs to be constrained by early.

Why?

​Per the handbook:

This keyword should only be applied by committers. The keyword signals that the ticket is a priority, and should be handled early in the next release cycle.

The high priority level flags that it needs to be addressed high in the bugfix queue.

But early keyword has a time restriction that the bugfix must be completed early alpha release cycle, else it will be punted out of the release.

For this bugfix, I'm seeing a high risk to users or the project to drive needing it committed early in alpha. Is there a high risk?

IMO this could have been fixed up until RC1. But since it was flagged as an early candidate, once the early phase of the alpha cycle was past, it has to be punted.

To avoid punting it in 6.3, I suggest removing early but leaving the high priority. With this approach, the high priority bugfix can happen at any time between now and 6.3 RC1.

@audrasjb do you agree?

[petitphp]

In WP 6.2.1 a change was made to stop processing shortcodes in block templates.

This change is being discussed after some issues were reported for some site after the update to 6.2.1.

This ticket might become invalid if shortcode processing is removed, or the patch for this ticket will need to be updated to match the definitive solution.

[oglekler]

I tested with 6.3-alpha-56078-src and Twenty Twenty-Three theme, placing inline shortcode into Header, Footer and Post Meta templates part — it is working everywhere.

So, it looks like the initial issue needs to be tested again to see if it was fully fixed during 6.2.2.

[poena]

I can reproduce the issue with 6.3-beta3-56130-src.

I have used the provided classic/hybrid test theme with some additions like the default gallery short code. I found that short codes are not rendered in the header part.

[mai21]

Test Report

Reproducible with the following steps based on https://core.trac.wordpress.org/ticket/56780#comment:14

1- Install and activate the following theme
https://core.trac.wordpress.org/attachment/ticket/56780/test-56780_updated.zip

2- Create a post and add short code

<!-- wp:shortcode -->
	[hello_world]
	<!-- /wp:shortcode -->

3- View the frontend of the post

Environment

• OS: Ubuntu 23.04
• Web Server: Nginx
• PHP: 8.1.9
• WordPress: 6.3-beta4-56228)
• Browser: Chrome 114.0.5735.198
• Theme: [test-56780_updated.zip]
• Active Plugins:
  • Core rollback 1.3.5
  • WordPress Beta Tester 3.5.2

Actual Results

Additional Notes

• Expected o/p:

masthead: hello_world!

Short code in post edit

hello_world!

Supplemental Artifacts

• Tested with copy past the changes here into related file ​https://github.com/WordPress/wordpress-develop/pull/3851 and can see it is fixed

[mai21]

updated theme

[mai21]

after patch#3851

[mai21]

before path#3851

[oglekler]

According to @mai21, this ticket was fixed on the related PR.

[audrasjb]

As per today's bug scrub, and since WP 6.3 release candidate 1 is planned tomorrow, let's move this ticket to milestone 6.4 wirth early keyword, so it doesn't wait until the end of the beta cycle to be committed.

[hellofromTonya]

I've removed the early keyword. Why? See my comments above.

​The handbook explains the keyword as:

This keyword should only be applied by committers. The keyword signals that the ticket is a priority, and should be handled early in the next release cycle.

early means the commits need to happen during the early part of the Alpha cycle.

Though it would be great to prepare the work for commit as early as possible, commits for this ticket should not be constrained to only happen during the early Alpha phase. Rather, defect commits can happen up to 6.4 RC 1.

To aid in 6.4 tracking and scrubs, I've removed the early keyword.

[deargrumpycats]

Is there a chance in the next update to include Bard AI? I recently run a

​Entertainment website and it would be
perfect if we can have AI. Thanks.

[petitphp]

Test Report

This report validates that the indicated patch addresses the issue.

Patch tested: ​https://github.com/WordPress/wordpress-develop/pull/3851

Environment

• OS: macOS 13.4.1
• Web Server: Nginx
• PHP: 7.4.33
• WordPress: 6.4-alpha-56267-src
• Browser: Firefox 118.b2
• Theme: Theme: test-56780 v0.1.0 (same as comment:14) with custom masthead.html content ​from ironprogrammer's gist
• Active Plugins: ø

Actual Results

• ✅ do_shortcode (original reported issue)
• ✅ wptexturize (e.g. processed smart quotes)
• ✅ convert_smilies
• ✅ shortcode_unautop
• ✅ wp_filter_content_tags (added decoding="async" loading="lazy" attributes to <img> tag)
• ✅ $wp_embed->autoembed

Additional Notes

• I was able to reproduce the original issue in trunk.
• Prior to doing the testing, the patch has been refresh with latest change up until ​trunk@3f9bc4e.

Supplemental Artifacts

See attachments bellow for screenshots before and after applying the patch.

[petitphp]

Screenshot of a single post with block template part before patch 3851

[petitphp]

Screenshot of a single post with block template part after patch 3851

[iammehedi1]

Test Report

This report validates that the indicated patch fixed the issue.

Patch tested: ​https://github.com/WordPress/wordpress-develop/pull/3851

Environment

• OS: Windows 11 Pro (22H2)
• Web Server: nginx/1.25.1
• PHP: 7.4.33
• WordPress: 6.4-alpha-56267-src
• Browser: Chrome Version 116.0.5845.141 (Official Build) (64-bit)
• Theme: Test 56780 - https://core.trac.wordpress.org/attachment/ticket/56780/test-56780_updated.zip.
• No Active Plugins

Prior to Applying the Patch

• Followed the provided guidelines of @mai21
• ❌ The shortcode remained unprocessed within the template section, whether it was placed within a <!-- wp:shortcode --> block or used independently.

After Applying The Patch

• ✅ The issue was fixed.

Supplemental Artifacts

Before patch:

After Patch:

[zunaid321]

Test Report

This report validates that the indicated patch addresses the issue.

Patch tested: ​https://github.com/WordPress/wordpress-develop/pull/3851

Environment

• OS: Windows 11 (22H2)
• Web Server: nginx/1.25.1
• PHP: 7.4.33
• WordPress: 6.4-alpha-56267-src
• Browser: Chrome Version 113.0.5672.126 (Official Build) (64-bit)
• Theme: [test-56780_updated.zip]
• Plugins: None activated

Before Applying The Patch

• Followed the instructions of @mai21
• ❌ shortcode was not applied.

After Applying The Patch

• Followed the same instructions.
• ✅ shortcode was applied properly.

Supplemental Artifacts

Before: ​https://i.imgur.com/GsMts5M.png
After: ​https://i.imgur.com/T2KG8sU.png

[marybaum]

@costdev has asked @SergeyBiryukov to run the patch against the test suite. If nothing breaks, we're ready to commit. @Clorith has also volunteered to help.

[nicolefurlan]

@costdev was testing for this patch successful? Just checking to make sure that this will be ready for inclusion in 6.4, as RC1 is in ~2 weeks. Thanks :)

[costdev]

Hi @nicolefurlan, this still needs to be checked against the security test suite. It's too close to 6.4 Beta 2 to get this done now, so I'll get this done before 6.4 Beta 3.

[costdev]

Update:

• The PR passes against the security tests.
• I've made some updates to the PR to document the $wp_embed global, improve test method docblocks, names and use get_echo() for brevity.
• I've pinged @spacedmonkey to see if some performance checks are possible to ensure that this PR doesn't cause too much of impact.

This one should still be landable in 6.4, provided that the performance checks don't show a drastic impact.

[spacedmonkey]

As this may have effects on performance, I have added the performance focus so the performance team can support with benchmarks and profiles.

I think this PR introduces some subtle security issues. For example an anonymous user can use a shortcode in a comment to render private data in the frontend. (comments being rendered by a block comments block)

I think this PR introduces some subtle security issues. For example an anonymous user can use a shortcode in a comment to render private data in the frontend. (comments being rendered by a block comments block)

I created a test shortcode and adding to comment text. I am not seeing the shortcode render there.

The template part block does something like this:

	// Run through the actions that are typically taken on the_content.
	$content                       = shortcode_unautop( $content );
	$content                       = do_shortcode( $content );
	$seen_ids[ $template_part_id ] = true;
	$content                       = do_blocks( $content );
	unset( $seen_ids[ $template_part_id ] );
	$content = wptexturize( $content );
	$content = convert_smilies( $content );
	$content = wp_filter_content_tags( $content, "template_part_{$area}" );

	// Handle embeds for block template parts.
	global $wp_embed;
	$content = $wp_embed->autoembed( $content );

I wonder if there's a way to reuse that code in both places, one could use the other or rely on a common function. (It might require the changes to be made in the Gutenberg repository)

I created a test shortcode and adding to comment text. I am not seeing the shortcode render there.

Did you use a theme that uses the function above to render a "template part" that contains the comments block?

+1 render_block_core_template_part and block_template_part should use the same logic. It is confusing why they do not.

[spacedmonkey]

Is there an example theme that uses this function to help test?

[costdev]

Update:

• There are ​security concerns about the PR. This needs some additional testing.
• To help with this testing, there is ​a request for a test theme that uses block_template_part() to render a template part containing a comments block.
  • @pbiron can you help with this bit?
• There is ​a request for feedback from @flixos90 regarding wp_filter_content_tags().
• There is ​a suggestion to abstract the logic for shared usage between render_block_core_template_part() and block_template_part() which may require changes in the Gutenberg repository.
  • @youknowriad Are you happy to follow up on your suggestion with a new issue on the GitHub repository? If so, can you drop a link to the issue in here as a reference when it's created?

This ticket needs more work and 6.4 Beta is scheduled for release tomorrow. To give us time to explore and resolve the above, I'm milestoning this ticket for 6.5.

[pbiron]

Replying to costdev:

• To help with this testing, there is ​a request for a test theme that uses block_template_part() to render a template part containing a comments block.
  • @pbiron can you help with this bit?

A template part containing a comments block? No, I don't have one. But the test-56780.zip​ test theme attached to this ticket should be modifiable to have a comments block.

But since I've never built a site (in 13 years of WP development) that allows commenting I'm don't think I'm the best person to create a "realistic" test theme with that feature ;-)

[youknowriad]

Good summary @costdev there are two small issues that force us to adopt a different strategy:

• The block_template_part function is now maintained in core and can't be modified in Gutenberg.
• The template part block can't use any code that only lives in the Gutenberg plugin (because it's auto imported to core later)
• The Gutenberg plugin supports two WP versions.

So the way we should go about this is the following:

• Update block_template_part on Core to duplicate what the template part block does (while ensuring the function or the extracted code can be reused in the template part block).
• We'd have to wait that the function has been released since two WP releases before making the change in the template part block in the Gutenberg plugin to reuse the function in the template part block. The other approach would be to use a Gutenberg specific function as a fallback in the meantime something like:

<?php
if ( function_exists( 'block_template_part' ) ) {
   gutenberg_block_template_part();
} else {
  block_template_part();
}

[joemcgill]

I'm removing the performance focus, since this ticket is not primarily about addressing a performance concern. Even so, @costdev feel free to reach out to folks on the team if we can support in any way.

[tastynifty]

Any chance in the next update to include AI module such as bard or gemini? I'm new to wp, I build a ​Blog and I guessed it would be awesome if we can have AI inside wp/theme. It will make the job easier. Thanks.

[hellofromTonya]

Restoring this ticket to be a bug report.

@tastynifty this ticket is tracking a specific bug report for shortcode blocks not being expanded (their render callbacks are not called) in a classic theme when using block-based template part. Discussions for adding AI module or capabilities should happen in a different Trac ticket. If one is not currently open, feel free to open a new feature request Trac ticket.

[poena]

#60444 was marked as a duplicate.

[hmbashar]

Test Report

Patch tested: ​https://github.com/WordPress/wordpress-develop/pull/3851

Environment

• WordPress: 6.6-alpha-57778-src
• PHP: 8.3.7
• Server: nginx/1.25.4
• Database: mysqli (Server: 8.3.0 / Client: mysqlnd 8.3.7)
• Browser: Chrome 125.0.0.0
• OS: macOS
• Theme: Test 56780 0.1.0
• MU Plugins: None activated
• Plugins:
  • Test Reports 1.1.0

Actual Results

Followed @mai21's instructions before and after applying the patch, but I see no changes.

​https://i.ibb.co/yp1c1Py/Before-Patch-Apply.png
​https://i.ibb.co/v1dsNgf/Screenshot-at-Jun-06-1-40-34-AM.png

@petitphp Thanks for the PR. Can you please check the conflicts :pray:

[nazmul111]

I could not find any proper scenario of what about need to be tested.

[desrosj]

Unfortunately this one did not get the attention it needs during the 6.6 cycle, so I'm going to punt.

I'm adding the early tag as I think this would benefit from a greater period of testing in trunk to confirm there are no unintended side effects somewhere.

[chaion07]

We reviewed this ticket during a recent bug-scrub session. @costdev do you still want to continue owning this Ticket? If so are you able to share a status update? Thanks.

Are the existing Test Results still valid? If not, can we have contributors conduct updated tests? We believe this Ticket needs a bit more attention. Cheers!

Props to @mukesh27 for the discussion

[joemcgill]

During Dev Chat today, we reviewed this ticket and concluded that this should be punted. Given that this has rolled for a few releases without much traction, I'm setting it as Future Release rather than allowing it to continue rolling to future release milestones without momentum. If it picks up traction, it can be moved back to a milestone.

[ecairol]

For folks out there still having this issue, here's a workaround while the patch makes it to core.

At the top of your template, extract the block HTML to a variable and apply do_shortcode():

<?php
ob_start();
block_template_part( 'header' );
$block_header = do_shortcode( ob_get_clean() );

Make sure to include wp_head() after these lines, so that the proper CSS inline styles are embedded on the screen:

<?php
<head>
  <?php wp_head(); ?>

And then echo $block_header to actually render the block where you want it.

[sppramodh]

Test Report

Description

This report validates whether the indicated patch works as expected.

Patch tested: ​https://github.com/WordPress/wordpress-develop/pull/3851

Environment

• WordPress: 6.7-RC2
• PHP: 8.2.18
• Server: Apache/2.4.59 (Win64) PHP/8.2.18 mod_fcgid/2.3.10-dev
• Database: mysqli (Server: 8.3.0 / Client: mysqlnd 8.2.18)
• Browser: Firefox 132.0
• OS: Windows 10/11
• Theme: Test 56780 0.1.0
• MU Plugins: None activated
• Plugins:
  • Secure Custom Fields 6.3.10.2
  • Test Reports 1.1.0
  • WooCommerce 9.3.3
  • WordPress Beta Tester 3.6.1

Actual Results

1. ❌ The patch dose not solve the issue completely.

Additional Notes

• As noted in comment #50, there was concern that the patch might introduce subtle security issues by allowing a shortcode to be used within comments.
• As instructed in comment #54, I used the test theme and modified it to use block_template_part() to render a template part containing comments block.
• After applying the patch, the shortcode is now processed in the comments weather short code is added in the comment or using <!-- wp:shortcode --> in the comment template part, which, as anticipated in comment #50, could introduce security issues. Therefore, the patch needs further improvement.

Supplemental Artifacts

Screenshot: