preg_replace_callback in do_shortcode returns empty for large posts

[AaronCampbell]

This is definitely related to #6877, though I don't know that I'd call it a dupe. Anyway, the problem is that on long posts the ​pcre.backtrack_limit is exceeded. I have a post that is 105k+ characters and I couldn't process even a single shortcode.

Setting the pcre.backtrack_limit to 1,000,000 worked (ini_set('pcre.backtrack_limit', 1000000);), but the default (100,000) should definitely work.

Here are some thoughts to kick around:

• I wonder if it would help to break the post into chunks first: process from the first [ to the last ] for shortcodes then re-add the start and end. In many cases this would reduce the amount of text being processed.
• Alternatively, maybe some of them could be handled with str_replace? (if [shortcode] exists and [/shortcode] doesn't). I know that this would only work for shortcodes with no attributes or content, but if that's a common usage (in the specific case I was working with, all the shortcodes were like this).

Anyway, the database allows 4,294,967,296 characters for a post, but I run against this problem pretty regularly on posts over 100,000.

[DD32]

Setting the pcre.backtrack_limit to 1,000,000 worked (ini_set('pcre.backtrack_limit', 1000000);)

(Just for anyone else who comes accross this ticket, That requires PHP 5.2+)

[AaronCampbell]

Good point. Also, I'm noticing that a lot of my tickets have been commented on and I never receive E-Mails. What happened to the notification?

[mrmist]

Some sample ​text for reproduction if required. (Fixed by the workaround). Looking at this you can see when it's going to happen because the word count stays at 0.

[Denis-de-Bernardy]

this bracktrack stuff is soooo annoying. not to mention the complete lack of error messages that comes with it. it really makes one wonder why they added pcre into php. I mean, php5.2 renders pcre completely useless except for the most basic patterns.

In case it helps, I ended up fixing this on my end by doing less work in each regexp. That is, instead of matching exactly what I needed, and catching everything I needed, all in one go, I ended up using multiple calls to preg_match, preg_replace, and preg_replace_callback...

and yes, that's another way of saying gobbling more php resources to work around a limit on resources imposed by php. :D

[Denis-de-Bernardy]

"fixing this on my end" was for my plugins, of course.

[Denis-de-Bernardy]

Please try the attached fix. It splits shortcode workflow in two steps -- first fetch shortcodes, then match parameters.

[mrmist]

I applied the patch but it did not fix the issue for me (my sample text attached above still appears as a blank post.)

[mrmist]

Edit - I should say it's in the post editor where it appears blank (after pressing save / publish.)

[tmcookies]

is it perhaps a problem of mysql's longtext-implementation? I have the same problem here. When i write a long article i can't really save it. I've tried copying the whole text into the mysql-table directly and didn't work either!

[Denis-de-Bernardy]

odd... when I tried using your sample text, it worked fine, even with the multiple shortcodes I inserted into it.

for the save that's not working, it's likely another bug. could it be a plugin you've installed or something?

[mrmist]

Replying to Denis-de-Bernardy:

for the save that's not working, it's likely another bug. could it be a plugin you've installed or something?

I was testing against trunk from the svn with the patch applied, so the install doesn't have any active plugins.

Just to clarify when you were able to save the sample text successfully was that with or without an increased pcre.backtrack_limit?

[Denis-de-Bernardy]

With a higher pcre backtrace limit. Further investigation reveals the issue with your test text comes from wpautop.

[Denis-de-Bernardy]

there is a bug opened for wpautop: #6877

[mrmist]

Replying to Denis-de-Bernardy:

With a higher pcre backtrace limit. Further investigation reveals the issue with your test text comes from wpautop.

OK. Keep in mind though that if I increase the limit then my sample text goes through fine, without any patch. It's the default limit that poses a problem.

[Denis-de-Bernardy]

yaya, but please try the patch in #6877 - I already requested that it be added 5.7.1, but I (sadly) have no commit rights over here. :-)

[mrmist]

Replying to Denis-de-Bernardy:

yaya, but please try the patch in #6877 - I already requested that it be added 5.7.1, but I (sadly) have no commit rights over here. :-)

I'm not sure which patch you mean. The patch attached to #6877 was committed way back in the 2.5 release, and still seems to be present in 10404. (So in other words doesn't fix the sample text.)

[Denis-de-Bernardy]

this one:

​http://trac.wordpress.org/attachment/ticket/6877/6877.2.diff

It was committed only a few weeks ago, but only into wp 2.8 -- not wp 2.7.x

[mrmist]

My local test follows trunk so yeah, it has that patch. It's not a fix for the sample text above.

I feel I should point out that I don't have this issue myself, the sample text was from an old thread on the support forums, but it serves to illustrate.

[azaozz]

Replying to Denis-de-Bernardy:

this one:

​http://trac.wordpress.org/attachment/ticket/6877/6877.2.diff

It was committed only a few weeks ago, but only into wp 2.8 -- not wp 2.7.x

Actually it was committed about 3 months ago to 2.7 (then trunk) [9255].

[tmcookies]

guys, could those of you for which the text doesn't work perhaps try inserting the text directly using phpmyadmin? For me even that doesn't work but if i upload a file with the content embedded in a sql-insert-statement and "mysql -e file" then it works. Could it be a $_POST-thing?

[Denis-de-Bernardy]

definitely not. see my comment on #6877 on the line that makes wpautop clunk on this piece of text.

[mrmist]

Replying to tmcookies:

guys, could those of you for which the text doesn't work perhaps try inserting the text directly using phpmyadmin?

Done. Inserted fine through phpmyadmin.

Still blank when viewed from edit post.

[tmcookies]

sorry. I just found out what the problem in my case was. For anyone else having the same problem as i have: i had the php-module suhosin enabled. This has some "max"-options enabled (mainly suhosin.request.max_value_length and suhosin.post.max_value_length. Those limit the size of the value php gets from $_POST and $_GET.. So, either disable suhosin or change these values..

[azaozz]

(In [10527]) Reduce backtracking in autop, fixes #6877, see #8553

[xibe]

Sorry to burst any potential bubble here, but as far as I can tell it is still not fixed in the current trunk.

I told ​the long story on the support forums, but I'll make it short here: using 2.7, I couldn't only get ​my loooong post to work by installing the Text Control plugin and setting it to nl2br rather than wpautop. I tested the same text (50 Kb) in a trunk install, and it would fail all the same.

I'd be happy to do more tests if I can help this get fixed proper.

If I'm wrong anywhere in my assumptions, please let me know. Also, if this would be a better fit in a re-opened #6877, please tell me, and I'll oblige.

Thank you.

[hakre]

an idea on this one: shortcodes aren't that complex, it should be possible to write a parser that does it with php functions (instead of regex) and therefore shouldn't run into such problems.

can be used transparent through the current api.

the current docblocks in there are a bit unclear wether or not it is possible to put shortcodes into shortcodes. ​http://codex.wordpress.org/Shortcode_API states yes, but they won't be handeled automatically. ​http://svn.automattic.com/wordpress-tests/wp-testcase/test_shortcode.php states that the same shortcode can enclose itself but left open where the closing code is.

[Denis-de-Bernardy]

Could anyone with looong posts try the attached patch?

[xibe]

I tested the patch on my trunk install, using the ​aforementioned long post of mine: ​it's not working.

One thing I don't remember noticing before, is that the post gets blank if saved in Visual mode - saving in HTML mode still leaves the post intact, it's just that online version that's blank.

---

Testing with a huge, generated ​Lorem Ispum, the post does get properly published (2.8-bleeding w/ patch AND 2.7.2). But if fails if you add just one [caption]-ed image...

[Denis-de-Bernardy]

patch needs to be refreshed. moving to future.

[Denis-de-Bernardy]

see also: #9264

[Brusdeylins]

Hi,

the resolution is descibed on my website (in german):
​http://www.brusdeylins.info/wordpress/probleme-mit-shortcodes/

The problem is in shortcodes.php in the function get_shortcode_regex().
Here the RegEx has a non-greedy part between the brackets, which should exclude the closing bracket. This would reduce the memory usage of the backtracking process in the RegEx-Engine (the reason of this problem).

My solution for the last line of this function looks like this (WordPress 2.8):
return '(.?)\[('.$tagregexp.')\b([^{\]]*?)(\/)?\](?:(.+?)\[\/\2\])?(.?)';}

Here I also removed the non-catching brackets around the catching brackets around the slash after the non-greedy area... Don't know why these are existed...

[Brusdeylins]

Here my modified function in shortcodes.php:

function get_shortcode_regex() {
	global $shortcode_tags;
	$tagnames = array_keys($shortcode_tags);
	$tagregexp = join( '|', array_map('preg_quote', $tagnames) );

	//return '(.?)\[('.$tagregexp.')\b(.*?)(?:(\/))?\](?:(.+?)\[\/\2\])?(.?)';
	  return '(.?)\[('.$tagregexp.')\b([^\]]*?)(\/)?\](?:(.+?)\[\/\2\])?(.?)';
	
}

[Denis-de-Bernardy]

@Brusdeylins - please try the alternative patch over at #9264

[Brusdeylins]

Hi,

Bug 9264 is NOT THE SOLUTION ! (of course, it is an other topic...)
I tested it on my long article "Yahoo Finance API"
​http://www.brusdeylins.info/projects/yahoo-finance-api/
(This article contains source codes with square brackets in it, you have to face this!)

In the Attachment in 9264 you are still running over the first closing bracket of the first shortcode tag: (.*?)
This don't make sence if your shortcode names should not contain any brackets... So replace the (.*?) like described above...

About 9264:
the (\/)? allows you to use shortcode tags like [NAME/] at the beginning. And with (?:(.+?)\[\/\2\])? the closing tag is optional. So you can use tags combinations like:

[x]...[/x] 
[x/]...[/x] 
[x]
[x/]

[Denis-de-Bernardy]

Adding a /x param to make things more readable... The current one is:

"/
(.?)                # optionally catch an extra bracket used to escape shortcode
\[($tagregexp)\b    # begin shortcode
  (.*?)             # optional attributes (non-greedy, stops on close bracket)
  (?:(\/))?         # optional slash to indicate a closing tag, not always set
  \]                # closing bracket
(?:                 # optional content and closing shortcode
  (.+?)             # the content
  \[\/\2\]          # close shortcode with a back reference
)?
(.?)                # optionally catch an extra bracket used to escape shortcode
/x"

you're suggesting:

"/
(.?)                # optionally catch an extra bracket used to escape shortcode
\[($tagregexp)\b    # begin shortcode
  ([^\]]*?)         # optional attributes (non-greedy), equivalent to using a dot
  (\/)?             # optional slash to indicate a closing tag, always set
  \]                # closing bracket
(?:                 # optional content and closing shortcode
  (.+?)             # the content
  \[\/\2\]          # close shortcode with a back reference
)?
(.?)                # optionally catch an extra bracket used to escape shortcode
/x"

the suggestion in #9264 is the following:

"/
(.?)                # optionally catch an extra bracket used to escape shortcode
\[($tagregexp)\b    # begin shortcode
  (.*?)             # optional attributes (non-greedy, stops on close bracket)
  (?:(\/))?         # optional slash to indicate a closing tag, not always set
  \]                # closing bracket
(
  ?(4)              # stop if we already have 4 back references
                    # it would mean we've a slash bracket
  |                 # else
  (?:               # optional content and closing shortcode
    (.+?)           # the content
    \[\/\2\]        # close shortcode with a back reference
  )?
)?
(.?)                # optionally catch an extra bracket used to escape shortcode
/x"

A few notes:

• Using your patch would remove the possibility for the optimization in #9264 (since we'd always have.
• Your attribute-related regex might be a bit more efficient from a performance standpoint.

Personally, I was hoping that the optimization in the other ticket would allow you to do [foo/] in order to optimize your way through the mess.

All of these patches, however, leave a few fundamental issues behind:

1. Something like [foo[bar] get treated as a foo shortcode with [bar as an attribute

2. We can get brackets in a shortcode's content, which can potentially lead to nested shortcodes:

[foo]bar[][/foo]
[foo][bar/][/foo]

3. A non-closed shortcode can gobble up an entire document before the regex bails and decides that it has no closing shortcode:

[foo]
long document that runs into the backtrack limit starts here

It's the two last points which I believe are troubling. I take it that some users actually want brackets in shortcodes, so fixing point 2 probably isn't an option. Bailing after, say, 1000 characters after the start of the content might be a good bet.

Would you mind attaching your post's source in a text file, exactly as it is in the WP editor, and specify the plugins you're using?

[Brusdeylins]

working fix

[Brusdeylins]

a long article from brusdeylins.info for fix in 8553.3.diff

[Brusdeylins]

OK, in my last post I wrote "This don't make sence if your shortcode names should not contain any brackets". Here I ment the parameters of the shortcodes, not the names itself. Sorry… I want to explain, how I understand the described regular expression.

--- The double brackets ---

First the double brackets (one non-catching and one catching):

(?:(\/))?         # optional slash to indicate a closing tag, not always set

My suggestion: you don’t need the first non-catching brackets, because they don’t deactivate the catching brackets inside and they don’t catch more then the brackets inside. They are only producing more backtrackings! The Meaning of this part is: The result array holds on position 4 the slash or "empty", but always has the same amount of array elements! And the meaning of “(\/)?” is the same, because the question mark is outside of both brackets…

Here an code example with double brackets:

$TXT = “abcdefg”;
$pattern = '/.*(?:(cd)).*/';
preg_match_all($pattern, $TXT, $array);
echo '<pre>', print_r($array, true), '</pre>';

And the result:

Array
(
    [0] => Array
        (
            [0] => abcdefg
        )

    [1] => Array
        (
            [0] => cd
        )
)

If you run the same program with the following regex:

$pattern = '/.*(cd).*/';

you get exact the same result:

Array
(
    [0] => Array
        (
            [0] => abcdefg
        )

    [1] => Array
        (
            [0] => cd
        )

)

So I think you can reduce this part of the regular expression. Or do you have another example where you get different results?

--- The memory problem with non-greedy quantifiers ---

(.*?)             # optional attributes (non-greedy, stops on close bracket)

Here I don’t think that my solution is equivalent to “using a dot”. Not if you have limits in memory (and time)! And here is the problem. PHP uses a “traditional NFA RegEx-Engine”. This means (in case of non-greedy quantifiers) that the engine likes “save states” and backtracking… if we use non-greedy quantifiers, we have a LIFO process… (This is what I learned yeas ago… maybe this means: trying all - returning the shortest one - if the memory can hold all temp. results?)

Try it. Replace the line like described in 8553.3.diff and you will see, that the example post will appear. With this fix, you don’t resolve your problem No. 3, as you wrote (there is still the pattern (.+?) in the expression)! I am using the WordPress plug-in “NextGen Gallery”. This plug-in uses shortcode tags like “[singlepic id=37 w=150 h=400 float=right]”. Here we don’t have (self) closing tags…

I attached the diff and the HTML-Code of the article I wrote. Happy debugging :-)

[Denis-de-Bernardy]

some more testdata:

[foo attr]bar[/foo]
[foo attr]bar[/foo][foo attr]bar[/foo]
[foo attr]bar[/foo][foo attr/]bar[/foo]
[foo attr]bar[/foo][foo attr/]bar
[foo attr]bar[/foo][foo attr]bar
[foo attr]bar[/foo][[foo attr]bar[/foo]
[foo attr]bar[/foo][[foo attr/]bar[/foo]
[foo attr]bar[/foo][[foo attr/]bar
[foo attr]bar[/foo][[foo attr]bar
[foo attr]bar[/foo][foo attr]bar[/foo]]
[foo attr]bar[/foo][foo attr/]]bar[/foo]
[foo attr]bar[/foo][foo attr/]bar]
[foo attr]bar[/foo][foo attr]bar]
[foo attr]bar[/foo][[foo attr]bar[/foo]]
[foo attr]bar[/foo][[foo attr/]]bar[/foo]
[foo attr]bar[/foo][[foo attr/]]bar
[foo attr]bar[/foo][[foo attr]]bar
[foo attr/]bar[/foo]
[foo attr/]bar[/foo][foo attr]bar[/foo]
[foo attr/]bar[/foo][foo attr/]bar[/foo]
[foo attr/]bar[/foo][foo attr/]bar
[foo attr/]bar[/foo][foo attr]bar
[foo attr/]bar[/foo][[foo attr]bar[/foo]
[foo attr/]bar[/foo][[foo attr/]bar[/foo]
[foo attr/]bar[/foo][[foo attr/]bar
[foo attr/]bar[/foo][[foo attr]bar
[foo attr/]bar[/foo][foo attr]bar[/foo]]
[foo attr/]bar[/foo][foo attr/]]bar[/foo]
[foo attr/]bar[/foo][foo attr/]bar]
[foo attr/]bar[/foo][foo attr]bar]
[foo attr/]bar[/foo][[foo attr]bar[/foo]]
[foo attr/]bar[/foo][[foo attr/]]bar[/foo]
[foo attr/]bar[/foo][[foo attr/]]bar
[foo attr/]bar[/foo][[foo attr]]bar
[foo attr/]bar
[foo attr/]bar[foo attr]bar[/foo]
[foo attr/]bar[foo attr/]bar[/foo]
[foo attr/]bar[foo attr/]bar
[foo attr/]bar[foo attr]bar
[foo attr/]bar[[foo attr]bar[/foo]
[foo attr/]bar[[foo attr/]bar[/foo]
[foo attr/]bar[[foo attr/]bar
[foo attr/]bar[[foo attr]bar
[foo attr/]bar[foo attr]bar[/foo]]
[foo attr/]bar[foo attr/]]bar[/foo]
[foo attr/]bar[foo attr/]bar]
[foo attr/]bar[foo attr]bar]
[foo attr/]bar[[foo attr]bar[/foo]]
[foo attr/]bar[[foo attr/]]bar[/foo]
[foo attr/]bar[[foo attr/]]bar
[foo attr/]bar[[foo attr]]bar
[foo attr]bar
[foo attr]bar[foo attr]bar[/foo]
[foo attr]bar[foo attr/]bar[/foo]
[foo attr]bar[foo attr/]bar
[foo attr]bar[foo attr]bar
[foo attr]bar[[foo attr]bar[/foo]
[foo attr]bar[[foo attr/]bar[/foo]
[foo attr]bar[[foo attr/]bar
[foo attr]bar[[foo attr]bar
[foo attr]bar[foo attr]bar[/foo]]
[foo attr]bar[foo attr/]]bar[/foo]
[foo attr]bar[foo attr/]bar]
[foo attr]bar[foo attr]bar]
[foo attr]bar[[foo attr]bar[/foo]]
[foo attr]bar[[foo attr/]]bar[/foo]
[foo attr]bar[[foo attr/]]bar
[foo attr]bar[[foo attr]]bar
[[foo attr]bar[/foo]
[[foo attr]bar[/foo][foo attr]bar[/foo]
[[foo attr]bar[/foo][foo attr/]bar[/foo]
[[foo attr]bar[/foo][foo attr/]bar
[[foo attr]bar[/foo][foo attr]bar
[[foo attr]bar[/foo][[foo attr]bar[/foo]
[[foo attr]bar[/foo][[foo attr/]bar[/foo]
[[foo attr]bar[/foo][[foo attr/]bar
[[foo attr]bar[/foo][[foo attr]bar
[[foo attr]bar[/foo][foo attr]bar[/foo]]
[[foo attr]bar[/foo][foo attr/]]bar[/foo]
[[foo attr]bar[/foo][foo attr/]bar]
[[foo attr]bar[/foo][foo attr]bar]
[[foo attr]bar[/foo][[foo attr]bar[/foo]]
[[foo attr]bar[/foo][[foo attr/]]bar[/foo]
[[foo attr]bar[/foo][[foo attr/]]bar
[[foo attr]bar[/foo][[foo attr]]bar
[[foo attr/]bar[/foo]
[[foo attr/]bar[/foo][foo attr]bar[/foo]
[[foo attr/]bar[/foo][foo attr/]bar[/foo]
[[foo attr/]bar[/foo][foo attr/]bar
[[foo attr/]bar[/foo][foo attr]bar
[[foo attr/]bar[/foo][[foo attr]bar[/foo]
[[foo attr/]bar[/foo][[foo attr/]bar[/foo]
[[foo attr/]bar[/foo][[foo attr/]bar
[[foo attr/]bar[/foo][[foo attr]bar
[[foo attr/]bar[/foo][foo attr]bar[/foo]]
[[foo attr/]bar[/foo][foo attr/]]bar[/foo]
[[foo attr/]bar[/foo][foo attr/]bar]
[[foo attr/]bar[/foo][foo attr]bar]
[[foo attr/]bar[/foo][[foo attr]bar[/foo]]
[[foo attr/]bar[/foo][[foo attr/]]bar[/foo]
[[foo attr/]bar[/foo][[foo attr/]]bar
[[foo attr/]bar[/foo][[foo attr]]bar
[[foo attr/]bar
[[foo attr/]bar[foo attr]bar[/foo]
[[foo attr/]bar[foo attr/]bar[/foo]
[[foo attr/]bar[foo attr/]bar
[[foo attr/]bar[foo attr]bar
[[foo attr/]bar[[foo attr]bar[/foo]
[[foo attr/]bar[[foo attr/]bar[/foo]
[[foo attr/]bar[[foo attr/]bar
[[foo attr/]bar[[foo attr]bar
[[foo attr/]bar[foo attr]bar[/foo]]
[[foo attr/]bar[foo attr/]]bar[/foo]
[[foo attr/]bar[foo attr/]bar]
[[foo attr/]bar[foo attr]bar]
[[foo attr/]bar[[foo attr]bar[/foo]]
[[foo attr/]bar[[foo attr/]]bar[/foo]
[[foo attr/]bar[[foo attr/]]bar
[[foo attr/]bar[[foo attr]]bar
[[foo attr]bar
[[foo attr]bar[foo attr]bar[/foo]
[[foo attr]bar[foo attr/]bar[/foo]
[[foo attr]bar[foo attr/]bar
[[foo attr]bar[foo attr]bar
[[foo attr]bar[[foo attr]bar[/foo]
[[foo attr]bar[[foo attr/]bar[/foo]
[[foo attr]bar[[foo attr/]bar
[[foo attr]bar[[foo attr]bar
[[foo attr]bar[foo attr]bar[/foo]]
[[foo attr]bar[foo attr/]]bar[/foo]
[[foo attr]bar[foo attr/]bar]
[[foo attr]bar[foo attr]bar]
[[foo attr]bar[[foo attr]bar[/foo]]
[[foo attr]bar[[foo attr/]]bar[/foo]
[[foo attr]bar[[foo attr/]]bar
[[foo attr]bar[[foo attr]]bar
[foo attr]bar[/foo]]
[foo attr]bar[/foo]][foo attr]bar[/foo]
[foo attr]bar[/foo]][foo attr/]bar[/foo]
[foo attr]bar[/foo]][foo attr/]bar
[foo attr]bar[/foo]][foo attr]bar
[foo attr]bar[/foo]][[foo attr]bar[/foo]
[foo attr]bar[/foo]][[foo attr/]bar[/foo]
[foo attr]bar[/foo]][[foo attr/]bar
[foo attr]bar[/foo]][[foo attr]bar
[foo attr]bar[/foo]][foo attr]bar[/foo]]
[foo attr]bar[/foo]][foo attr/]]bar[/foo]
[foo attr]bar[/foo]][foo attr/]bar]
[foo attr]bar[/foo]][foo attr]bar]
[foo attr]bar[/foo]][[foo attr]bar[/foo]]
[foo attr]bar[/foo]][[foo attr/]]bar[/foo]
[foo attr]bar[/foo]][[foo attr/]]bar
[foo attr]bar[/foo]][[foo attr]]bar
[foo attr/]]bar[/foo]
[foo attr/]]bar[/foo][foo attr]bar[/foo]
[foo attr/]]bar[/foo][foo attr/]bar[/foo]
[foo attr/]]bar[/foo][foo attr/]bar
[foo attr/]]bar[/foo][foo attr]bar
[foo attr/]]bar[/foo][[foo attr]bar[/foo]
[foo attr/]]bar[/foo][[foo attr/]bar[/foo]
[foo attr/]]bar[/foo][[foo attr/]bar
[foo attr/]]bar[/foo][[foo attr]bar
[foo attr/]]bar[/foo][foo attr]bar[/foo]]
[foo attr/]]bar[/foo][foo attr/]]bar[/foo]
[foo attr/]]bar[/foo][foo attr/]bar]
[foo attr/]]bar[/foo][foo attr]bar]
[foo attr/]]bar[/foo][[foo attr]bar[/foo]]
[foo attr/]]bar[/foo][[foo attr/]]bar[/foo]
[foo attr/]]bar[/foo][[foo attr/]]bar
[foo attr/]]bar[/foo][[foo attr]]bar
[foo attr/]bar]
[foo attr/]bar][foo attr]bar[/foo]
[foo attr/]bar][foo attr/]bar[/foo]
[foo attr/]bar][foo attr/]bar
[foo attr/]bar][foo attr]bar
[foo attr/]bar][[foo attr]bar[/foo]
[foo attr/]bar][[foo attr/]bar[/foo]
[foo attr/]bar][[foo attr/]bar
[foo attr/]bar][[foo attr]bar
[foo attr/]bar][foo attr]bar[/foo]]
[foo attr/]bar][foo attr/]]bar[/foo]
[foo attr/]bar][foo attr/]bar]
[foo attr/]bar][foo attr]bar]
[foo attr/]bar][[foo attr]bar[/foo]]
[foo attr/]bar][[foo attr/]]bar[/foo]
[foo attr/]bar][[foo attr/]]bar
[foo attr/]bar][[foo attr]]bar
[foo attr]bar]
[foo attr]bar][foo attr]bar[/foo]
[foo attr]bar][foo attr/]bar[/foo]
[foo attr]bar][foo attr/]bar
[foo attr]bar][foo attr]bar
[foo attr]bar][[foo attr]bar[/foo]
[foo attr]bar][[foo attr/]bar[/foo]
[foo attr]bar][[foo attr/]bar
[foo attr]bar][[foo attr]bar
[foo attr]bar][foo attr]bar[/foo]]
[foo attr]bar][foo attr/]]bar[/foo]
[foo attr]bar][foo attr/]bar]
[foo attr]bar][foo attr]bar]
[foo attr]bar][[foo attr]bar[/foo]]
[foo attr]bar][[foo attr/]]bar[/foo]
[foo attr]bar][[foo attr/]]bar
[foo attr]bar][[foo attr]]bar
[[foo attr]bar[/foo]]
[[foo attr]bar[/foo]][foo attr]bar[/foo]
[[foo attr]bar[/foo]][foo attr/]bar[/foo]
[[foo attr]bar[/foo]][foo attr/]bar
[[foo attr]bar[/foo]][foo attr]bar
[[foo attr]bar[/foo]][[foo attr]bar[/foo]
[[foo attr]bar[/foo]][[foo attr/]bar[/foo]
[[foo attr]bar[/foo]][[foo attr/]bar
[[foo attr]bar[/foo]][[foo attr]bar
[[foo attr]bar[/foo]][foo attr]bar[/foo]]
[[foo attr]bar[/foo]][foo attr/]]bar[/foo]
[[foo attr]bar[/foo]][foo attr/]bar]
[[foo attr]bar[/foo]][foo attr]bar]
[[foo attr]bar[/foo]][[foo attr]bar[/foo]]
[[foo attr]bar[/foo]][[foo attr/]]bar[/foo]
[[foo attr]bar[/foo]][[foo attr/]]bar
[[foo attr]bar[/foo]][[foo attr]]bar
[[foo attr/]]bar[/foo]
[[foo attr/]]bar[/foo][foo attr]bar[/foo]
[[foo attr/]]bar[/foo][foo attr/]bar[/foo]
[[foo attr/]]bar[/foo][foo attr/]bar
[[foo attr/]]bar[/foo][foo attr]bar
[[foo attr/]]bar[/foo][[foo attr]bar[/foo]
[[foo attr/]]bar[/foo][[foo attr/]bar[/foo]
[[foo attr/]]bar[/foo][[foo attr/]bar
[[foo attr/]]bar[/foo][[foo attr]bar
[[foo attr/]]bar[/foo][foo attr]bar[/foo]]
[[foo attr/]]bar[/foo][foo attr/]]bar[/foo]
[[foo attr/]]bar[/foo][foo attr/]bar]
[[foo attr/]]bar[/foo][foo attr]bar]
[[foo attr/]]bar[/foo][[foo attr]bar[/foo]]
[[foo attr/]]bar[/foo][[foo attr/]]bar[/foo]
[[foo attr/]]bar[/foo][[foo attr/]]bar
[[foo attr/]]bar[/foo][[foo attr]]bar
[[foo attr/]]bar
[[foo attr/]]bar[foo attr]bar[/foo]
[[foo attr/]]bar[foo attr/]bar[/foo]
[[foo attr/]]bar[foo attr/]bar
[[foo attr/]]bar[foo attr]bar
[[foo attr/]]bar[[foo attr]bar[/foo]
[[foo attr/]]bar[[foo attr/]bar[/foo]
[[foo attr/]]bar[[foo attr/]bar
[[foo attr/]]bar[[foo attr]bar
[[foo attr/]]bar[foo attr]bar[/foo]]
[[foo attr/]]bar[foo attr/]]bar[/foo]
[[foo attr/]]bar[foo attr/]bar]
[[foo attr/]]bar[foo attr]bar]
[[foo attr/]]bar[[foo attr]bar[/foo]]
[[foo attr/]]bar[[foo attr/]]bar[/foo]
[[foo attr/]]bar[[foo attr/]]bar
[[foo attr/]]bar[[foo attr]]bar
[[foo attr]]bar
[[foo attr]]bar[foo attr]bar[/foo]
[[foo attr]]bar[foo attr/]bar[/foo]
[[foo attr]]bar[foo attr/]bar
[[foo attr]]bar[foo attr]bar
[[foo attr]]bar[[foo attr]bar[/foo]
[[foo attr]]bar[[foo attr/]bar[/foo]
[[foo attr]]bar[[foo attr/]bar
[[foo attr]]bar[[foo attr]bar
[[foo attr]]bar[foo attr]bar[/foo]]
[[foo attr]]bar[foo attr/]]bar[/foo]
[[foo attr]]bar[foo attr/]bar]
[[foo attr]]bar[foo attr]bar]
[[foo attr]]bar[[foo attr]bar[/foo]]
[[foo attr]]bar[[foo attr/]]bar[/foo]
[[foo attr]]bar[[foo attr/]]bar
[[foo attr]]bar[[foo attr]]bar

[Denis-de-Bernardy]

Brusdeylins: be so kind to try the patch we'll attach to #10082 when it's ready. I'll be based on the following tests:

​http://core.trac.wordpress.org/attachment/ticket/10082/shortcode-tests.php

[Denis-de-Bernardy]

see ​http://core.trac.wordpress.org/attachment/ticket/10082/10082.diff

[hakre]

The test-data should be extended to reflect multi-line shortcodes as well. according to some reports, there are issues as well and AFAIK regexes are aware of line endings (when commanded). Therefore looks reasonable to test that as well.

[Brusdeylins]

Hi all,

I tested the Patch #10082, but it don't fix my problem.
My long article don't appear with this DIFF.

Sorry.
Changed back to my solution :)

[xibe]

Sorry I didn't notice earlier there was a patch for this.
I applied it to my current trunk, and in my case it does work!

Before applying patch:

• post updated in HTML mode would display fine in both editor and online
• same post updated in Visual mode would become blank

After applying patch:

• In both case, the post does get published, and is still editable in Visual mode after clicking the update button.

For the sake of helping other test my conditions, I'm adding it to Trac, zipped. In current 2.9 trunk, copy the code into the HTML editor, publish & check online presence ; switch to Visual editor, update & witness blankness both in editor and online.

Note: I don't seem to have an issue with ​Brusdeylins' test post, even without applying the patch, which sounds weird to me.

[xibe]

xibe's long post

[jamescollins]

We have encountered this on several WordPress installations after upgrading PHP from v5.1.6 to 5.2.10.

The large pages (which contain various shortcodes) have no output displayed on the page when it is viewed.

Our pcre.backtrack_limit is set to the default value (100000).

Adding the following to wp-config.php allows the pages to be parsed successfully.

ini_set('pcre.backtrack_limit', 1000000);

Although I'm not sure of the side effects of increasing this value.

[hakre]

Side effects are probably an increase of memory usage and/or CPU load.

[usermrpapa]

We too have several users who have encountered this bug using shortcodes on posts/pages with a lot of content (or images)... increasing the backtrack limit does mitigate the problem (at least for now), but it would be great if this could be optimized to not require the increased limit...

[Brusdeylins]

This Problem still exists in WP 2.8.6.
My solution in ./wp-includes/shortcodes.php :

function get_shortcode_regex() {
  global $shortcode_tags;
  $tagnames = array_keys($shortcode_tags);
  $tagregexp = join( '|', array_map('preg_quote', $tagnames) );
  
  // defect for long posts
  // return '(.?)\[('.$tagregexp.')\b(.*?)(?:(\/))?\](?:(.+?)\[\/\2\])?(.?)'; 
	
  // solution for brusdeylins.info
  return '(.?)\[('.$tagregexp.')\b([^\]]*?)(\/)?\](?:(.+?)\[\/\1\])?(.?)'; 	
}

[jalenack]

Replying to Brusdeylins:

This Problem still exists in WP 2.8.6.
My solution in ./wp-includes/shortcodes.php :

function get_shortcode_regex() {
  global $shortcode_tags;
  $tagnames = array_keys($shortcode_tags);
  $tagregexp = join( '|', array_map('preg_quote', $tagnames) );
  
  // defect for long posts
  // return '(.?)\[('.$tagregexp.')\b(.*?)(?:(\/))?\](?:(.+?)\[\/\2\])?(.?)'; 
	
  // solution for brusdeylins.info
  return '(.?)\[('.$tagregexp.')\b([^\]]*?)(\/)?\](?:(.+?)\[\/\1\])?(.?)'; 	
}

I was suffering from this issue in 2.8.x and 2.9 on several posts (7600 words with images etc). I added the pcre ini_set('pcre.backtrack_limit', 1000000); and that fixed the problem. Wordpress was taking a really long time to save so I tried using the above get_shortcode_regex() code, and it broke my [caption] shortcode that was generated by visual mode:

[caption id="attachment_9522" align="alignright" width="350" caption="___desc__"]<a href="__url__"><img class="size-full wp-image-9522" title="___desc___" src="____url___" alt="___desc___" width="350" height="351" /></a>[/caption]

broke as in failed to parse and left a hanging [/caption]. I didn't try any other patches besides the pcre.backtrack_limit part, which did fix my problem.

[mrmist]

I didn't try any other patches besides the pcre.backtrack_limit part, which did fix my problem.

Yeah but increasing the backtrack limit is not an actual fix, just a workaround. This ticket has twisted on for a while now but I reckon there is stuff in here that could probably fix it.

[nacin]

No activity for months. Punting.

[Denis-de-Bernardy]

Looks like this is occurring on wordpress.com....

See: ​http://hakre.wordpress.com/

[aaroncampbell]

It looks like all his posts are coming back as empty. They can't all be that big, right?

[aaroncampbell]

Well, now they're all showing, and none of them seem anywhere near big enough to have triggered this bug.

[nacin]

It was an unrelated wp.com bug.

[mrmist]

I was inspired to check this out again with the sample text I posted 22 months ago, and the other sample text added to the ticket and it all now worksforme on stock 3.0.1 and trunk. (Without patch)

[msafi]

I think ini_set('pcre.backtrack_limit', 1000000); can also be set in functions.php of the theme or a plugin file—it doesn't have to be in wp-config.php.

I set mine to 200000 instead of 1000000 and it did the trick.

How come there's no fix for this yet? It wasn't easy tracing down the source of this problem!

[mdawaffe]

Is this a dupe of #15600?

If not, is this still a bug as of [18952]?

[aaroncampbell]

Yes, it looks #15600 fixed get_shortcode_regex() which is where the issue was for this ticket. I don't actually have a test case for this one at the moment, but I think it was fixed. I'm fine with closing at this point.

[xibe]

Well, according to my ​aforementioned long post (using the latest nightly), this is still an issue in some cases.

[aaroncampbell]

Long post content with gallery and caption shortcodes

[aaroncampbell]

Replying to xibe:

Well, according to my ​aforementioned long post (using the latest nightly), this is still an issue in some cases.

Are you sure it's the same problem (pcre.backtrack_limit)?

I created a test post​ that uses the gallery and caption shortcodes (about 1500-1600 shortcodes total). I added the post to a ​3.2.1 install and a ​trunk install. On the trunk install it displays as expected and on the 3.2.1 install it shows as blank (which is what was happening when this ticket was filed).

I think this is fixed.

[xibe]

You're very right, I mixed the issues in my head, sorry :/