wptexturize messes up shortcode with parameter meta_compare=">="

[bobbingwide]

This is a direct follow on to #28564.

I have a number of shortcodes where I pass args to get_posts() almost directly from the shortcode.

In one particular example I use meta_compare="<" to display posts which are earlier than a certain date.
In another I use meta_compare=">=" to show posts on or greater than the date.

In WordPress 4.0 the ">=" meta_compare has stopped working.

Expected result: WordPress 3.9.1

[shortcode meta_compare=">=" ]

will end up passing a meta_compare value of >= to get_posts(). i.e.

[meta_compare] = >=

Unexpected result: WordPress 4.0

the array field value passed to get_posts() becomes

[meta_compare] => ’>=’

This problem is caused by wptexturize(), which converts a double quote to &#8221; or a single quote to &#8217;

Pragmatic workaround:

 remove_filter( "the_content", "wptexturize" );

Alternatively, I could introduce a specific test on the meta_compare parameter value and strip unexpected entities.
I may have to do this anyway since I've seen a situation where the visual editor has converted > to &gt; which has caused a similar failure.

[miqrogroove]

There is a possible compromise here. Since it looks like we may have to import the registered shortcode names for #29557 anyway, it may not be necessary to filter the contents of the shortcodes at all. The only test that fails in this case is the "Let's get crazy" test from test_tag_avoidance(). That particular test allowed recursive nesting of square braces within a hyperlink element within a shortcode. I'm guessing that case is less valuable than the one described above in the ticket.

[miqrogroove]

It looks like we can use this patch if there are no other problems reported:

https://core.trac.wordpress.org/attachment/ticket/29557/miqro-29557.5.patch

[miqrogroove]

Duplicate of #29557.

Going to wrap these all into one ticket because they have the same patch.

[miqrogroove]

See #29661 for discussion about future versions.

[bobbingwide]

Just to confirm that the patch has fixed the problem that I reported.

[bobbingwide]

The fix for #29557 in 4.0.1 has not fixed this problem.

Actually the problem has got worse.

It no longer accepts
[shortcode meta_compare="<"]

My pragmatic workaround is still needed.
I am reopening this defect since it actually occurs in a live site.

[miqrogroove]

Yes we are aware of the problem. It is documented here:

​http://codex.wordpress.org/Shortcode_API#HTML

I've done some brainstorming here:

​http://www.miqrogroove.com/blog/2014/shortcode-problems-resolved/

The fixes in 4.0.1 were a time-consuming effort and since 4.1 beta is out already we are looking at 4.2 for a full solution.

[miqrogroove]

I should also caution that there might not be a solution to using meta_compare=">=". We might end up fixing #15694 to provide better escaping functions and then conclude the raw angle braces are a wontfix idea. It depends heavily on design and backward compatibility decisions being made in other areas. To fix one thing we have to break another, etc.

[bobbingwide]

OK, I have reviewed miqrogroove's blog and appended my comments, which were to agree on implementing the filters using different priorities.

I have developed a local patch that

1. implements the priority change
2. supports ‘no_texturize_shortcodes’
3. and replaces wptexturize with a new filter function that

a) detects no texturize tags created around shortcodes which don’t like having their output texturized AND
b) calls a stripped down version that doesn’t do any regex processing on shortcodes…

It is currently a combination of one of my plugins and some changes in formatting.php

Changes are:

do_shortcode_tag() now wraps the result of a 'no_texturize_shortcodes' shortcode with
<!--notext:--> and <!--dotext:-->

the replacement for wptexturize() loops through the output finding notext:'s and dotext:'s

• stuff before a notext: is passed to the stripped down texturize function
• everything between notext: and the next dotext: is left as is
• then the loop continues after the dotext:

In my solution I also reprioritize wpautop() - invoking it so that it doesn't convert newlines to <br />.
I don't call shortcode_unautop() either.

I'm now in the process of moving the plugin code into core so that I can run unit tests.
But I'm happy with what I see. My shortcodes work as they did in 3.9 and wptexturize logic works as well.

Another option that I discussed ( with johnbillion ) was to implement a solution that supports condition expressions. LT, LE, EQ, GE, GT etc... I can develop this solution for my own code, OR it could be implemented in WP_Query. Comments?

[miqrogroove]

It's something that might get traction during the 4.2 cycle or might not. I will help work on this where I can, but after the 4.1 release at the earliest.

[boonebgorges]

Another option that I discussed ( with johnbillion ) was to implement a solution that supports condition expressions. LT, LE, EQ, GE, GT etc... I can develop this solution for my own code, OR it could be implemented in WP_Query. Comments?

I'd be happy to talk about this for 4.2 or beyond. A separate enhancemet ticket might be appropriate.

[bobbingwide]

OK, will raise separate enhancement ticket for condition expressions.

[miqrogroove]

#29658 was marked as a duplicate.

[miqrogroove]

Can we just make this a duplicate of #15694? At this point they are both solved by proper escaping.

[miqrogroove]

Duplicate of #15694.

I've updated the Summary on the other ticket to include "<" and similar characters. Based on extensive testing and feedback from several core committers, I am confident that we cannot support raw angle braces inside of shortcodes in future versions and the best solution will be to implement escaping functions for all shortcode input. This includes administrators who have unfiltered_html capability. We are generally in agreement that past versions unfortunately created the impression that shortcodes were magical in the HTML context, when actually they were not and their attributes need to be HTML encoded for bugs-free operation. While this might create some minor inconveniences for those people editing raw HTML, the larger consideration is that this direction will fix several problems for users of shortcode GUI such as image galleries, image captions, and other media.