Shortcode regex no longer matches [shortcode=XXX]

[kraftbj]

After [34747], a shortcode in the format of [shortcode=xxx] is no longer rendered.

In reviewing the Shortcode Roadmap posts, I'm not sure if breaking this format was intended or not.

Discovered as it breaks one of the forms of Jetpack's YouTube shortcode: ​https://github.com/Automattic/jetpack/issues/3121

[Ipstenu]

Possibly related.

This no longer works:

[video mp4="link" wbem="link"]

it outputs the code directly onto the page without processing. BUT.... This does work:

[video mp4="link"][video wbem="link"]

Not hard to fix for me but it's unexpected. Also this works fine in the visual editor, just not on display.

Edit to add that this works.

[video width="640" height="360" mp4="LINK"]

[kraftbj]

#34979 was marked as a duplicate.

[azurecurve]

I have this same issue with at least two of my plugins (azurecurve-bbcode and azurecurve-flags).

I'll need to review other plugins and see which might be affected as I have a few with shortcodes that may work similar to this.

[smashballoon]

I have 2 plugins which use shortcodes and have had several reports of issues with shortcode options since the 4.4 update. I'm not sure whether or not this is related, but we received a ticket from a user today who is experiencing issues after updating to 4.4 which may be useful:

"It seems the wordpress editor occasionally replaces standard 0x020 space characters with 0x0a0 (unicode non-breaking space). I haven't been able to isolate exactly what causes this, probably some weird transition between the text and visual modes during editing. But it sometimes causes the plugin to gag on the shortcodes.
Took me two days to figure this out! :) A working and non-working short code look identical in the wordpress editor but if I copy them over to a hex editor, I can see the non-working one has a 0x0a0 instead of a 0x020 space character between the shortcode name and the parameter."

It looks like some other plugins are also having similar problems too: ​https://wordpress.org/support/topic/shortcode-broken-with-wp-44-upgrade

[Ipstenu]

I can confirm this is a weird non-space space.

I copied the code to another product and it gave a proper error on it, saying there was an unreadable char between mp4="link" and wbem="link"

In the editor, it was just a space.

Well ain't that special?

[kraftbj]

Another notable thing. At least with Jetpack's YouTube shortcode. If you have the broken style in a post by itself, it fails. If you put the broken style, followed by one with a space, both of them work.

Fails:

[youtube=https://youtu.be/IGrqiSYc1oM&w=600&h=440]

Both are rendered:

[youtube=https://youtu.be/IGrqiSYc1oM&w=600&h=440]
[youtube url=https://youtu.be/IGrqiSYc1oM&w=600&h=440]

[darrenbrettking]

In addition to the YouTube shortcode issues, I noticed issues with other themes I use as well - including this one: ​http://demo.themeum.com/#starter

Note: you won't see the issue on their live demo because they haven't moved to 4.4 yet, but I saw it on some of the sites I'm hosting.

Question: larger picture, can someone help me understand how this relates to the WP commitment to backwards compatibility? My understanding is that it's up to developers to update the PHP code to handle the shortcode issue, right? Just wondering how that fits with the overall commitment to sites not breaking with updates. Thanks!

[darrenbrettking]

Interestingly, some of this seems to be a parsing/caching/syncing issue - rather than an actual conflict; at least in certain situations. I went in and just re-typed some of my shortcodes - without changing anything - and after I re-saved the page everything rendered as it should. That's the only change I made - I just manually retyped the shortcodes. Note that copy-and-pasting won't work. You need to type it by hand. But for me I didn't even have to retype whole sections - just some of the shortcode, and even later shortcodes that I didn't retype then rendered correctly.

[azurecurve]

The original problem reported was with shortcodes like [url=www.bbc.co.uk]BBC[/url]

[azurecurve]

The original problem reported was with shortcodes like [url=www.bbc.co.uk]

[patrick@…]

Hello all,

We are having this problem on two different sites, two different plugins.

A quick google search for "wordpress 4.4 shortcodes broken" reveals at least others having reported this on various forums for various plugin shortcodes.

Does someone have a handle on this problem and working on a fix?

Is there a workaround for this until WP 4.4 can be patched?

Thanks!

[dd32]

This is a fun issue..

Has [shortcode=var] ever actually been documented as a "thing"? What about [shortcode/var]? or even [shortcode!var]? both of which previously worked. (I'm mostly wondering where that syntax came from originally, I suspect it was accidental and "just worked" with the lazy parser we had)

[shortcode/var] works in 4.4 still, mostly by accident AFAICT

What this is caused by is this bit of code in do_shortcode() which detects what shortcodes are in the content:

// Find all registered tag names in $content. 
preg_match_all( '@\[([^<>&/\[\]\x00-\x20]++)@', $content, $matches ); 

That causes [shortcode=foo to be recognised as shortcode=foo being the shortcode name, but [shortcode&foo, [shortcode/foo, or [shortcode foo is understood to only be shortcode.

Pinging @miqrogroove can you step in here and clarify what the expected case is here? Should = be a reserved character that can't be contained in a shortcode name too? are there others that should potentially be added too? All punctuation?

[kraftbj]

Replying to dd32:

(I'm mostly wondering where that syntax came from originally, I suspect it was accidental and "just worked" with the lazy parser we had)

The backhistory I was able to pull up. The oldest reference I saw of this was from a WP.com commit from January 2006 (575) when we added support for a [youtube=XXX], before the Shortcode API existed. It was a the_content filter that replaced this "pre-shortcode" into the full embed code. There seemed to be other pre-shortcodes in this style. I didn't research it much, but the few I saw were for adding embed code for services that no longer exist.

After the Shortcode API was built, WP.com dropped the filter and converted it to a true shortcode. I don't know if this format was part of the thought when making the Shortcode API or not, so not sure if this working was a happy accident or an intentional (albeit maybe undocumented) feature.

Worth noting that one of the oldest examples of shortcodes noted in the Codex was this plugin: ​https://wordpress.org/plugins/bbcode/ by Alex Mills, which was linked in the ​Codex in June 2008 and still exists in the list of External Resources on that page as of now. One of the first examples in the plugin's description includes [url="http://wordpress.org/"]WordPress[/url]

[aaroncampbell]

Replying to dd32:

I'm pretty sure that [shortcode=var] working was a "happy accident", with the expected version of that being [shotcode var]. I don't think it was ever documented as working (but I certainly don't claim canonical knowledge of this) in any "official" documentation (like we had that back then!). Having said that, it was clearly discovered rather quickly...probably because of it's common usage in bbcode?

I think the best solution would be to add = to the list of reserved characters not allowed in a shortcode name. Personally I think that shortcode names should be [\w-], but I know @miqrogroove did a lot of research into what people are already using.

[dd32]

I think the best solution would be to add = to the list of reserved characters not allowed in a shortcode name.

That's the same conclusion I'd come to, and what I think we should move ahead with.
For 4.5 we should also change the warning string not to hard-code that list in the translation and instead use %s with the characters.

[jorbin]

@miqrogroove as the master of shortcodes, can you take a look?

[aaroncampbell]

34939.diff​ adds '=' to the list of reserved characters in the regex as well as the warning text. It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just '​http://wordpress.org'.

There is a new test added as well, but I noted in two different comments that the [shortcode=http://wordpress.org] format is NOT recommended (I'd love to see it go away some day, so less reliance on it would be a plus).

[Otto42]

Replying to Ipstenu:

I can confirm this is a weird non-space space.

I copied the code to another product and it gave a proper error on it, saying there was an unreadable char between mp4="link" and wbem="link"

In the editor, it was just a space.

Well ain't that special?

I believe that this falls into the realm of it used to work before because of the "happy accident".

TinyMCE has a habit of converting spaces (0x20) into non-breaking spaces (0xa0). It's done this for quite a while. Reproducing it is easy: type in a number of spaces in a row. Every other one will become a non-breaking space (0xa0).

The regular expression pointed out by @dd32 above does not recognize 0xa0 as valid separators. Seems like either 0xa0 should be added as a possible character not allowed in an shortcode/attribute name in order to prevent this issue, or the expressions reconsidered entirely, or just have a mass-replace on the shortcode as-a-whole to eliminate any 0xa0's by converting them to 0x20's.

[dd32]

Replying to aaroncampbell:

It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just '​http://wordpress.org'.

I'm not sure we can, or should, make that change here unfortunately. Existing shortcodes are probably relying upon it to work like that, I'd rather just leave that as an undocumented format

[aaroncampbell]

Replying to dd32:

Replying to aaroncampbell:

It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just '​http://wordpress.org'.

I'm not sure we can, or should, make that change here unfortunately. Existing shortcodes are probably relying upon it to work like that, I'd rather just leave that as an undocumented format

I think that $attributes[0] used to just be the url, but without the left trim before parsing them we end up with =https://wordpress.org instead. The trim will only affect the pre-parsed attribute string if the character immediately following the shortcode name is an equal sign. I'm pretty sure that's a safe move and also keeps parity with the way things used to work.

[dd32]

Replying to aaroncampbell:

Replying to dd32:

Replying to aaroncampbell:

It also trims '=' from the left side of the attributes string, so that the resulting attribute from [shortcode=http://wordpress.org] is just '​http://wordpress.org'.

I'm not sure we can, or should, make that change here unfortunately. Existing shortcodes are probably relying upon it to work like that, I'd rather just leave that as an undocumented format

I think that $attributes[0] used to just be the url, but without the left trim before parsing them we end up with =https://wordpress.org instead. The trim will only affect the pre-parsed attribute string if the character immediately following the shortcode name is an equal sign. I'm pretty sure that's a safe move and also keeps parity with the way things used to work.

That's not the case. It's always been passed as =attr. (At least according to /tags/4.2 & ​Jetpacks source)

[aaroncampbell]

Replying to dd32:

That's not the case. It's always been passed as =attr. (At least according to /tags/4.2 & ​Jetpacks source)

My bad. 34939.2.diff​ is updated to remove the ltrim.

[aaroncampbell]

34939.3.diff​ fixes the test to account for the = too.

[dd32]

In 36097:

Shortcodes: = is a reserved character in shortcode names, mark it as such.

This allows for shortcodes such as [shortcode=attribute] to work, which while never intentionally supported were widely used in the pre-shortcode days.

Props aaroncampbell.
Fixes #34939 for trunk.

[dd32]

In 36098:

Shortcodes: = is a reserved character in shortcode names, mark it as such.

This allows for shortcodes such as [shortcode=attribute] to work, which while never intentionally supported were widely used in the pre-shortcode days.

Merges [36097] to the 4.4 branch, minus a string change.
Props aaroncampbell.
Fixes #34939.

[David Findlay]

I'm having a problem that might be related to this. A shortcode I've got from a plugin works for logged in users, but not for non-logged in users. It just isn't drawing the content the shortcode refers to. Seems to only be an issue since recent updates.

[aaroncampbell]

Replying to David Findlay:

I'm having a problem that might be related to this. A shortcode I've got from a plugin works for logged in users, but not for non-logged in users. It just isn't drawing the content the shortcode refers to. Seems to only be an issue since recent updates.

Well, this was fixed in 4.4.1, so if you have 4.4.1 or later (you should be on 4.4.2 now), and it's still a problem, it wasn't this issue. Please make sure it's not a problem with the plugin itself, and if it is a core issue please open another ticket.

Thanks.