Make WordPress Core

Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#34939 closed defect (bug) (fixed)

Shortcode regex no longer matches [shortcode=XXX]

Reported by: kraftbj's profile kraftbj Owned by: miqrogroove's profile miqrogroove
Milestone: 4.4.1 Priority: normal
Severity: normal Version: 4.4
Component: Shortcodes Keywords:
Focuses: Cc:

Description

After [34747], a shortcode in the format of [shortcode=xxx] is no longer rendered.

In reviewing the Shortcode Roadmap posts, I'm not sure if breaking this format was intended or not.

Discovered as it breaks one of the forms of Jetpack's YouTube shortcode: https://github.com/Automattic/jetpack/issues/3121

Attachments (3)

34939.diff (3.0 KB) - added by aaroncampbell 9 years ago.
34939.2.diff (2.6 KB) - added by aaroncampbell 9 years ago.
34939.3.diff (2.6 KB) - added by aaroncampbell 9 years ago.

Download all attachments as: .zip

Change History (34)

#1 @Ipstenu
9 years ago

Possibly related.

This no longer works:

[video mp4="link" wbem="link"]

it outputs the code directly onto the page without processing. BUT.... This does work:

[video mp4="link"][video wbem="link"]

Not hard to fix for me but it's unexpected. Also this works fine in the visual editor, just not on display.

Version 2, edited 9 years ago by Ipstenu (previous) (next) (diff)

This ticket was mentioned in Slack in #core by jorbin. View the logs.


9 years ago

#3 @kraftbj
9 years ago

#34979 was marked as a duplicate.

#4 @SergeyBiryukov
9 years ago

  • Milestone changed from Awaiting Review to 4.4.1

#5 @azurecurve
9 years ago

I have this same issue with at least two of my plugins (azurecurve-bbcode and azurecurve-flags).

I'll need to review other plugins and see which might be affected as I have a few with shortcodes that may work similar to this.

#6 @smashballoon
9 years ago

I have 2 plugins which use shortcodes and have had several reports of issues with shortcode options since the 4.4 update. I'm not sure whether or not this is related, but we received a ticket from a user today who is experiencing issues after updating to 4.4 which may be useful:

"It seems the wordpress editor occasionally replaces standard 0x020 space characters with 0x0a0 (unicode non-breaking space). I haven't been able to isolate exactly what causes this, probably some weird transition between the text and visual modes during editing. But it sometimes causes the plugin to gag on the shortcodes.
Took me two days to figure this out! :) A working and non-working short code look identical in the wordpress editor but if I copy them over to a hex editor, I can see the non-working one has a 0x0a0 instead of a 0x020 space character between the shortcode name and the parameter."

It looks like some other plugins are also having similar problems too: https://wordpress.org/support/topic/shortcode-broken-with-wp-44-upgrade

#7 follow-up: @Ipstenu
9 years ago

I can confirm this is a weird non-space space.

I copied the code to another product and it gave a proper error on it, saying there was an unreadable char between mp4="link" and wbem="link"

In the editor, it was just a space.

Well ain't that special?

#8 @kraftbj
9 years ago

Another notable thing. At least with Jetpack's YouTube shortcode. If you have the broken style in a post by itself, it fails. If you put the broken style, followed by one with a space, both of them work.

Fails:

[youtube=https://youtu.be/IGrqiSYc1oM&w=600&h=440]

Both are rendered:

[youtube=https://youtu.be/IGrqiSYc1oM&w=600&h=440]
[youtube url=https://youtu.be/IGrqiSYc1oM&w=600&h=440]

#9 @darrenbrettking
9 years ago

In addition to the YouTube shortcode issues, I noticed issues with other themes I use as well - including this one: http://demo.themeum.com/#starter

Note: you won't see the issue on their live demo because they haven't moved to 4.4 yet, but I saw it on some of the sites I'm hosting.

Question: larger picture, can someone help me understand how this relates to the WP commitment to backwards compatibility? My understanding is that it's up to developers to update the PHP code to handle the shortcode issue, right? Just wondering how that fits with the overall commitment to sites not breaking with updates. Thanks!

#10 @darrenbrettking
9 years ago

Last edited 9 years ago by darrenbrettking (previous) (diff)

#11 @darrenbrettking
9 years ago

Interestingly, some of this seems to be a parsing/caching/syncing issue - rather than an actual conflict; at least in certain situations. I went in and just re-typed some of my shortcodes - without changing anything - and after I re-saved the page everything rendered as it should. That's the only change I made - I just manually retyped the shortcodes. Note that copy-and-pasting won't work. You need to type it by hand. But for me I didn't even have to retype whole sections - just some of the shortcode, and even later shortcodes that I didn't retype then rendered correctly.

Last edited 9 years ago by darrenbrettking (previous) (diff)

#12 @azurecurve
9 years ago

  • Version changed from 4.4 to trunk

The original problem reported was with shortcodes like [url=www.bbc.co.uk]BBC[/url]

Last edited 9 years ago by johnbillion (previous) (diff)

#13 @azurecurve
9 years ago

The original problem reported was with shortcodes like [url=www.bbc.co.uk]

Last edited 9 years ago by johnbillion (previous) (diff)

#14 @johnbillion
9 years ago

  • Keywords needs-patch added
  • Version changed from trunk to 4.4

#15 @patrick@…
9 years ago

Hello all,

We are having this problem on two different sites, two different plugins.

A quick google search for "wordpress 4.4 shortcodes broken" reveals at least others having reported this on various forums for various plugin shortcodes.

Does someone have a handle on this problem and working on a fix?

Is there a workaround for this until WP 4.4 can be patched?

Thanks!

#16 follow-ups: @dd32
9 years ago

This is a fun issue..

Has [shortcode=var] ever actually been documented as a "thing"? What about [shortcode/var]? or even [shortcode!var]? both of which previously worked. (I'm mostly wondering where that syntax came from originally, I suspect it was accidental and "just worked" with the lazy parser we had)

[shortcode/var] works in 4.4 still, mostly by accident AFAICT

What this is caused by is this bit of code in do_shortcode() which detects what shortcodes are in the content:

// Find all registered tag names in $content. 
preg_match_all( '@\[([^<>&/\[\]\x00-\x20]++)@', $content, $matches ); 

That causes [shortcode=foo to be recognised as shortcode=foo being the shortcode name, but [shortcode&foo, [shortcode/foo, or [shortcode foo is understood to only be shortcode.

Pinging @miqrogroove can you step in here and clarify what the expected case is here? Should = be a reserved character that can't be contained in a shortcode name too? are there others that should potentially be added too? All punctuation?

#17 in reply to: ↑ 16 @kraftbj
9 years ago

Replying to dd32:

(I'm mostly wondering where that syntax came from originally, I suspect it was accidental and "just worked" with the lazy parser we had)

The backhistory I was able to pull up. The oldest reference I saw of this was from a WP.com commit from January 2006 (575) when we added support for a [youtube=XXX], before the Shortcode API existed. It was a the_content filter that replaced this "pre-shortcode" into the full embed code. There seemed to be other pre-shortcodes in this style. I didn't research it much, but the few I saw were for adding embed code for services that no longer exist.

After the Shortcode API was built, WP.com dropped the filter and converted it to a true shortcode. I don't know if this format was part of the thought when making the Shortcode API or not, so not sure if this working was a happy accident or an intentional (albeit maybe undocumented) feature.

Worth noting that one of the oldest examples of shortcodes noted in the Codex was this plugin: https://wordpress.org/plugins/bbcode/ by Alex Mills, which was linked in the Codex in June 2008 and still exists in the list of External Resources on that page as of now. One of the first examples in the plugin's description includes [url="http://wordpress.org/"]WordPress[/url]

#18 in reply to: ↑ 16 @aaroncampbell
9 years ago

Replying to dd32:

I'm pretty sure that [shortcode=var] working was a "happy accident", with the expected version of that being [shotcode var]. I don't think it was ever documented as working (but I certainly don't claim canonical knowledge of this) in any "official" documentation (like we had that back then!). Having said that, it was clearly discovered rather quickly...probably because of it's common usage in bbcode?

I think the best solution would be to add = to the list of reserved characters not allowed in a shortcode name. Personally I think that shortcode names should be [\w-], but I know @miqrogroove did a lot of research into what people are already using.

#19 @dd32
9 years ago

I think the best solution would be to add = to the list of reserved characters not allowed in a shortcode name.

That's the same conclusion I'd come to, and what I think we should move ahead with.
For 4.5 we should also change the warning string not to hard-code that list in the translation and instead use %s with the characters.

#20 @jorbin
9 years ago

  • Owner set to miqrogroove
  • Status changed from new to assigned

@miqrogroove as the master of shortcodes, can you take a look?

@aaroncampbell
9 years ago

#21 follow-up: @aaroncampbell
9 years ago

  • Keywords has-patch needs-testing added; needs-patch removed

34939.diff adds '=' to the list of reserved characters in the regex as well as the warning text. It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just 'http://wordpress.org'.

There is a new test added as well, but I noted in two different comments that the [shortcode=http://wordpress.org] format is NOT recommended (I'd love to see it go away some day, so less reliance on it would be a plus).

#22 in reply to: ↑ 7 @Otto42
9 years ago

Replying to Ipstenu:

I can confirm this is a weird non-space space.

I copied the code to another product and it gave a proper error on it, saying there was an unreadable char between mp4="link" and wbem="link"

In the editor, it was just a space.

Well ain't that special?

I believe that this falls into the realm of it used to work before because of the "happy accident".

TinyMCE has a habit of converting spaces (0x20) into non-breaking spaces (0xa0). It's done this for quite a while. Reproducing it is easy: type in a number of spaces in a row. Every other one will become a non-breaking space (0xa0).

The regular expression pointed out by @dd32 above does not recognize 0xa0 as valid separators. Seems like either 0xa0 should be added as a possible character not allowed in an shortcode/attribute name in order to prevent this issue, or the expressions reconsidered entirely, or just have a mass-replace on the shortcode as-a-whole to eliminate any 0xa0's by converting them to 0x20's.

Last edited 9 years ago by Otto42 (previous) (diff)

#23 in reply to: ↑ 21 ; follow-up: @dd32
9 years ago

Replying to aaroncampbell:

It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just 'http://wordpress.org'.

I'm not sure we can, or should, make that change here unfortunately. Existing shortcodes are probably relying upon it to work like that, I'd rather just leave that as an undocumented format

#24 in reply to: ↑ 23 ; follow-up: @aaroncampbell
9 years ago

Replying to dd32:

Replying to aaroncampbell:

It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just 'http://wordpress.org'.

I'm not sure we can, or should, make that change here unfortunately. Existing shortcodes are probably relying upon it to work like that, I'd rather just leave that as an undocumented format

I think that $attributes[0] used to just be the url, but without the left trim before parsing them we end up with =https://wordpress.org instead. The trim will only affect the pre-parsed attribute string if the character immediately following the shortcode name is an equal sign. I'm pretty sure that's a safe move and also keeps parity with the way things used to work.

#25 in reply to: ↑ 24 ; follow-up: @dd32
9 years ago

Replying to aaroncampbell:

Replying to dd32:

Replying to aaroncampbell:

It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just 'http://wordpress.org'.

I'm not sure we can, or should, make that change here unfortunately. Existing shortcodes are probably relying upon it to work like that, I'd rather just leave that as an undocumented format

I think that $attributes[0] used to just be the url, but without the left trim before parsing them we end up with =https://wordpress.org instead. The trim will only affect the pre-parsed attribute string if the character immediately following the shortcode name is an equal sign. I'm pretty sure that's a safe move and also keeps parity with the way things used to work.

That's not the case. It's always been passed as =attr. (At least according to /tags/4.2 & Jetpacks source)

#26 in reply to: ↑ 25 @aaroncampbell
9 years ago

Replying to dd32:

That's not the case. It's always been passed as =attr. (At least according to /tags/4.2 & Jetpacks source)

My bad. 34939.2.diff is updated to remove the ltrim.

#27 @aaroncampbell
9 years ago

34939.3.diff fixes the test to account for the = too.

#28 @dd32
9 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 36097:

Shortcodes: = is a reserved character in shortcode names, mark it as such.

This allows for shortcodes such as [shortcode=attribute] to work, which while never intentionally supported were widely used in the pre-shortcode days.

Props aaroncampbell.
Fixes #34939 for trunk.

#29 @dd32
9 years ago

In 36098:

Shortcodes: = is a reserved character in shortcode names, mark it as such.

This allows for shortcodes such as [shortcode=attribute] to work, which while never intentionally supported were widely used in the pre-shortcode days.

Merges [36097] to the 4.4 branch, minus a string change.
Props aaroncampbell.
Fixes #34939.

#30 follow-up: @David Findlay
9 years ago

I'm having a problem that might be related to this. A shortcode I've got from a plugin works for logged in users, but not for non-logged in users. It just isn't drawing the content the shortcode refers to. Seems to only be an issue since recent updates.

#31 in reply to: ↑ 30 @aaroncampbell
9 years ago

  • Keywords has-patch needs-testing removed

Replying to David Findlay:

I'm having a problem that might be related to this. A shortcode I've got from a plugin works for logged in users, but not for non-logged in users. It just isn't drawing the content the shortcode refers to. Seems to only be an issue since recent updates.

This issue was fixed in 4.4.1, so if you have 4.4.1 or later (you should be on 4.4.2 now), and it's still a problem, it wasn't this. Please make sure it's not a problem with the plugin itself, and if it is a core issue please open another ticket.

Thanks.

Last edited 9 years ago by aaroncampbell (previous) (diff)
Note: See TracTickets for help on using tickets.