no_texturize_shortcodes filter broken in WP 5.9 beta 1 and 2

[pgn4web]

Hello, my wordpress plugin embed-chessboard uses the code below to avoid texturization of the text within a shortcode. In wordpess 5.9 beta 1 and 5.9 beta 2 the filter does not seems to have effect with texturization applied to the text within the shortcode (particularly straight quotes " get changed into opening/closing quotes, breaking things apart).

Has anything changes in wordpress 5.9 to break the code below? Or it's just a bug?

Thanks.

<?php
function embedchessboard_no_texturize( $shortcodes ) {
  $shortcodes[] = 'pgn';
  $shortcodes[] = 'pgn4web';
  return $shortcodes;
}

add_filter( 'no_texturize_shortcodes', 'embedchessboard_no_texturize' );

[hellofromTonya]

Hello @pgn4web,

Welcome back to WordPress Core's Trac!

There have been no changes to wptexturize() or the no_texturize_shortcodes filter in that function during the 5.9 cycle.

Can you provide more information please for contributors to investigate this?

• Did your shortcodes work on WordPress 5.8.2?
• Please deactivate all plugins, expect your the shortcode you're testing. Does this resolve the issue?
• Are you using a default bundled theme such as Twenty Twenty-Two (new block theme) or Twenty Twenty-One? If no, please switch to a bundled theme. Does this resolve the issue?
• If after testing above the issue remains, please provide the shortcode code and step-by-step instructions to reproduce the issue.

Thank you

[pgn4web]

Thanks @hellofromTonya for looking into this issue.

My shortcode comes from the embed-chessboard plugin, all details including full source code available on ​https://wordpress.org/plugins/embed-chessboard/

I originally coded and currently maintain the plugin. It works without any texturize related problems on the current release of wordpress. My demo site ​http://pgn4web-test.casaschi.net/wordpress/ is on wordpress 5.8.2, no issues there.

I tested the plugin against wordpress 5.9 beta 1 and beta 2. I did a new fresh install with no other plugin other than what comes with the wordpress download file and added my embed-chessboard. I used the default wordpress theme.

The shortcode code is relatively complex (it displays a chessboard with chess moves from chess moves text) however, long story short, if the text between the shortcode tags is like this:

[Event "World championship"]
[Site "Moscow URS"]
[Date "1985.10.15"]
[Round "16"]
[White "Karpov"]
[Black "Kasparov"]
[Result "0-1"]

1. e4 c5

then the plugin code receives this text instead, please note the change in the quotes characters (from " to “ and ”) that makes my plugin fail in showing the chess games properly:

[Event “World championship”]
[Site “Moscow URS”]
[Date “1985.10.15”]
[Round “16”]
[White “Karpov”]
[Black “Kasparov”]
[Result “0-1”]

1. e4 c5

I remember seeing this behaviour when originally coding the plugin; this was fixed by the following code in the plugin in order to avoid texturization of the text within the shortcode tags:

<?php
function embedchessboard_no_texturize( $shortcodes ) {
  $shortcodes[] = 'pgn';
  $shortcodes[] = 'pgn4web';
  return $shortcodes;
}

add_filter( 'no_texturize_shortcodes', 'embedchessboard_no_texturize' );

The code above works in wordpress 5.8.2 but does not seem to work in wordpress 5.9 beta 1 and beta 2. As a further proof, if I comment out on the 5.8.2 site the add_filter() instruction above, then the 5.8.2 site behaves exactly like the 5.9 betas. This confirms to me that the no_texturize_shortcodes filter in my plugin does not work with the 5.9 betas while it works with 5.8.2 and earlier.

If you'd like to reproduce the issue, you could:
1) install and activate the embed-chessboard plugin, it's on the wordpress plugin repository
2) create a new post, including the following shortcode:

[pgn initialHalfmove=16 autoplayMode=none]

[Event "World championship"]
[Site "Moscow URS"]
[Date "1985.10.15"]
[Round "16"]
[White "Karpov"]
[Black "Kasparov"]
[Result "0-1"]

1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nc6 5. Nb5 d6 
6. c4 Nf6 7. N1c3 a6 8.  Na3 d5 9. cxd5 exd5 10. exd5 Nb4 
11. Be2 Bc5 12. O-O O-O 13. Bf3 Bf5 14.  Bg5 Re8 15. Qd2 b5 
16. Rad1 Nd3 17. Nab1 h6 18. Bh4 b4 19. Na4 Bd6 20. Bg3  Rc8 
21. b3 g5 22. Bxd6 Qxd6 23. g3 Nd7 24. Bg2 Qf6 25. a3 a5 
26. axb4 axb4  27. Qa2 Bg6 28. d6 g4 29. Qd2 Kg7 30. f3 Qxd6 
31. fxg4 Qd4+ 32. Kh1 Nf6 33.  Rf4 Ne4 34. Qxd3 Nf2+ 
35. Rxf2 Bxd3 36. Rfd2 Qe3 37. Rxd3 Rc1 38. Nb2 Qf2 
39. Nd2 Rxd1+ 40. Nxd1 Re1+ 0-1

[/pgn]

You should see a chessboard but if you try clicking on the moves text the position of the chess pieces does not change. Compare for example with my demo site at ​http://pgn4web-test.casaschi.net/wordpress/

Hope this helps. Please let me know if you need any more help with this issue.

[pgn4web]

Some more information. I tested with wordpress 5.9 beta 3 and the problem is still there if I post a correct shortcode like this:

[pgn initialHalfmove=16 autoplayMode=none]

[Event "World championship"]
[Site "Moscow URS"]
[Date "1985.10.15"]
[Round "16"]
[White "Karpov"]
[Black "Kasparov"]
[Result "0-1"]

1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nc6 5. Nb5 d6 
6. c4 Nf6 7. N1c3 a6 8.  Na3 d5 9. cxd5 exd5 10. exd5 Nb4 
11. Be2 Bc5 12. O-O O-O 13. Bf3 Bf5 14.  Bg5 Re8 15. Qd2 b5 
16. Rad1 Nd3 17. Nab1 h6 18. Bh4 b4 19. Na4 Bd6 20. Bg3  Rc8 
21. b3 g5 22. Bxd6 Qxd6 23. g3 Nd7 24. Bg2 Qf6 25. a3 a5 
26. axb4 axb4  27. Qa2 Bg6 28. d6 g4 29. Qd2 Kg7 30. f3 Qxd6 
31. fxg4 Qd4+ 32. Kh1 Nf6 33.  Rf4 Ne4 34. Qxd3 Nf2+ 
35. Rxf2 Bxd3 36. Rfd2 Qe3 37. Rxd3 Rc1 38. Nb2 Qf2 
39. Nd2 Rxd1+ 40. Nxd1 Re1+ 0-1

[/pgn]

However, I noticed something really weird because of a typo. The shortcode works as expected with texturization if the opening shortcode tag is duplicated with an incorrect shortcode syntax, like this (notice the second [pgn] opening tag in the first line):

[pgn initialHalfmove=16 autoplayMode=none] [pgn]

[Event "World championship"]
[Site "Moscow URS"]
[Date "1985.10.15"]
[Round "16"]
[White "Karpov"]
[Black "Kasparov"]
[Result "0-1"]

1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nc6 5. Nb5 d6 
6. c4 Nf6 7. N1c3 a6 8.  Na3 d5 9. cxd5 exd5 10. exd5 Nb4 
11. Be2 Bc5 12. O-O O-O 13. Bf3 Bf5 14.  Bg5 Re8 15. Qd2 b5 
16. Rad1 Nd3 17. Nab1 h6 18. Bh4 b4 19. Na4 Bd6 20. Bg3  Rc8 
21. b3 g5 22. Bxd6 Qxd6 23. g3 Nd7 24. Bg2 Qf6 25. a3 a5 
26. axb4 axb4  27. Qa2 Bg6 28. d6 g4 29. Qd2 Kg7 30. f3 Qxd6 
31. fxg4 Qd4+ 32. Kh1 Nf6 33.  Rf4 Ne4 34. Qxd3 Nf2+ 
35. Rxf2 Bxd3 36. Rfd2 Qe3 37. Rxd3 Rc1 38. Nb2 Qf2 
39. Nd2 Rxd1+ 40. Nxd1 Re1+ 0-1

[/pgn]

This seems to me a confirmation that something odd is going on...

[pgn4web]

The issue is still present in wordpress 5.9 beta 4.

I've been able to narrow it down to an issue with the Twenty Twenty-Two theme.
With any earlier themes everything works fine, I tried Twenty Twenty-One, Twenty Twenty, Twenty Nineteen.

Only with the Twenty Twenty-Two theme the issue appears.

[audrasjb]

#54747 was marked as a duplicate.

[audrasjb]

Moving to 5.9 to make sure it's not an issue introduced in the current release cycle.

[hellofromTonya]

Test Report: Using the Embed Chessboard with TT1 theme: Left side is WP 5.8.2; Right side is WP 5.9 RC1. Not able to reproduce the reported issue.

[pgn4web]

Did you use the Twenty Twenty-Two theme?

I assume with TT1 you mean Twenty Twenty-One and that works fine with WP 5.9. It’s the new TT2 theme causing the issue.

[hellofromTonya]

Test Report: Embed Chessboard plugin + TT1 + WP 5.9 RC1 shown in Firefox, Safari, Chrome, and Edge browsers. Unable to reproduce reported issue.

[hellofromTonya]

Test Report

Env:

• OS: macOS Big Sur
• Localhost: Local
• WordPress: 5.8.2 and 5.9 RC1
• Theme: Twenty Twenty-One (TT1)
• Activated Plugins: Embed Chessboard, Beta Tester, Rollback Core
• Browser: Edge, Safari, Firefox, Chrome

Steps

• Step 1: Start with 5.8.2 and activate the TT1 theme
• Step 2: Add a post
• Step 3: Add a shortcode block
• Step 4: Copy the following shortcode and then paste into the shortcode block's field

  [pgn height=500 initialHalfmove=16 autoplayMode=none][Event "World championship"]
  [Site "Moscow URS"]
  [Date "1985.10.15"]
  [Round "16"]
  [White "Karpov"]
  [Black "Kasparov"]
  [Result "0-1"]
  
  1. e4 c5 2. Nf3 e6 3. d4 cxd4 4. Nxd4 Nc6 5. Nb5 d6 6.
  c4 Nf6 7. N1c3 a6 8. Na3 d5 9. cxd5 exd5 10. exd5 Nb4
  11. Be2 Bc5 12. O-O O-O 13. Bf3 Bf5 14. Bg5 Re8 15.
  Qd2 b5 16. Rad1 Nd3 17. Nab1 h6 18. Bh4 b4 19. Na4 Bd6
  20. Bg3 Rc8 21. b3 g5 22. Bxd6 Qxd6 23. g3 Nd7 24. Bg2
  Qf6 25. a3 a5 26. axb4 axb4 27. Qa2 Bg6 28. d6 g4 29.
  Qd2 Kg7 30. f3 Qxd6 31. fxg4 Qd4+ 32. Kh1 Nf6 33. Rf4
  Ne4 34. Qxd3 Nf2+ 35. Rxf2 Bxd3 36. Rfd2 Qe3 37. Rxd3
  Rc1 38. Nb2 Qf2 39. Nd2 Rxd1+ 40. Nxd1 Re1+ 0-1[/pgn]
  

• Step 5: Publish the post.
• Step 6: View the post.
• Step 7: Using the Beta Tester plugin, update to 5.9 RC1.
  • Go to Tools > Beta Testing.
  • Set to "Bleeding edge" and click "Save Changes" button.
  • Set to "Beta/RC Only" and then click the "Save Changes" button.
  • In the "Saved." admin message, click on the "Why don't you head on over and upgrade now." link or go to Dashboard > Updates.
  • Click on the button to install 5.9-RC1.
• Step 8: Open the post in a new browser tab.
• Step 9: Compare the chess game rendering between the 2 versions.

Results

• The 5.8.2 version of the game is identical to the 5.9-RC1 version of the game.
• No errors in the console.

The game renders identical in both 5.8.2 and 5.9-RC1 in multiple browsers.

Unable to reproduce the reported issue.

[hellofromTonya]

@pgn4web I'm unable to reproduce the reported issue as shown in the Test Report, test-report-59rc1-browsers.png​ and test-report-582-59RC1.png​. The game renders and interacts identically for me in both 5.8.2 and 5.9 RC1.

Can you please do a Test Report (follow the one I did to provide the exact env details and steps you take) and provide screenshots to show what you're seeing?

As I'm unable to reproduce the issue, moving this ticket back to Awaiting Review pending more information.

[pgn4web]

I’ll do a screenshot later today or tomorrow but please note that the Twenty Twenty-One theme works for me as well.

The problem occurs with the Twenty Twenty-Two theme.

[pgn4web]

See screenshot above, but keep in mind, this is using 5.9-RC1-52446 and the Twenty Twenty-Two theme.

[hellofromTonya]

Thanks for the update. Yes, I can reproduce in TT2 which is a block theme.

What's known:

• Nothing wptexturize() changed during the 5.9 cycle.
• Activating the latest Gutenberg plugin version (i.e. 12.3.0) does not resolve the issue.
• It works with non-block themes.
• It's isolated to block themes.

Some research:

• Seems to be isolated to FSE.
• I don't see anything in TT2 to cause this.
• There's an open issue in Gutenberg about single vs double quotes for shortcodes ​https://github.com/WordPress/gutenberg/issues/33813, though seems directly unrelated. Note: changing to single quotes does not resolve the issue.
• Removing the [Site...] content within the shortcode does not resolve the issue. (Checked to see if the [ and ] within were being interrupted incorrectly.)

There has been work due within the RichText component for sanitizing content within JS. A very quick look did not reveal to me how it's wired or if it's wired to the filter in wptexturize. This seems like a good place to start for deeper investigation by the editor team.

What's the next step? I'm thinking the next step towards resolution is to report it upstream in the Gutenberg repository. Why? It appears isolated to FSE / block theme environment. The investigation and fix will most likely be done there and then backported to Core with a package update (once the specific package is published).

@pgn4web Can you open a Bug Report ​issue here in Gutenberg? And then please share the issue URL here to bind the Trac ticket to the Gutenberg issue.

[pgn4web]

I can open a ticket, but I'll have an hard time to explain something I do not really understand myself, starting from why am I opening a Gutenberg ticket for the issue that I'm experiencing with WP 5.9 and the TT2 theme :-)

[pgn4web]

See ​https://github.com/WordPress/gutenberg/issues/37754

[hellofromTonya]

Thank you @pgn4web for opening the bug report in Gutenberg repo. I added more content to that report as well as commented on it too to help the team start where we are now in the investigation of the issue.

Discussions for the issue though should move to the Gutenberg issue ​https://github.com/WordPress/gutenberg/issues/37754. I'll move this ticket into 5.9.1 for tracking purposes.

[hellofromTonya]

On second thought, closing this ticket as reported upstream. Once resolved in Gutenberg, the fix will be backported to Core for inclusion in the next release. Hopefully that will happen for 5.9.1.

[pgn4web]

Does this mean that you expect the WP 5.9 release to show the bug with TT2?

I'll post a warning on my plugin site to avoid using TT2 because of a known bug.

[hellofromTonya]

Replying to pgn4web:

Does this mean that you expect the WP 5.9 release to show the bug with TT2?

I'll post a warning on my plugin site to avoid using TT2 because of a known bug.

The bug will exist in 5.9.0 and will be fixed in a minor when a fix is available.

For clarification, it's not a bug in the TT2 theme. Rather, it's potentially when texturize is called in relationship to when blocks render. That seems to be in the gutenberg_get_the_template_html() in Gutenberg which is the same as get_the_block_template_html()` in Core.

Tracking the issue's progress will happen here ​https://github.com/WordPress/gutenberg/issues/37754.

[pgn4web]

Hello,

the Gutenberg issue ​https://github.com/WordPress/gutenberg/issues/37754 does not show any progress several months after reporting; since then several new versions of WordPress.org and Gutenberg have been released.

Any clue why this has not been fixed yet?