Flooded 'wp_batch_split_terms' in cron

[array064]

After I upgrade Wordpress, from version 4.2.4 to version 4.3, there was an oddity: in the table PREFIX_options with option_name = 'cron', on option_value contains a lot of cron associated with 'wp_batch_split_terms' (see attachment I sent). This makes our site to be very very slow, especially on the back-end (wp-admin).

[dd32]

Hi @array064 Welcome to Trac :)

This is being tracked in #33423 and was fixed in [33646]

[array064]

Thank you for your quick response. :) So, while I am waiting for Wordpress version 4.3.1 which contains the fix, what should I do? Because cron in the table PREFIX_options progressively enlarged with the 'wp_batch_split_terms'.

[dd32]

@array064 You should fix it by apply this change to prevent it filling up https://core.trac.wordpress.org/changeset/33646/trunk/src/wp-includes/taxonomy.php you should then be safely able to delete the option to clear out the database row.

[Kent Brockman]

Replying to dd32:

@array064 You should fix it by apply this change to prevent it filling up https://core.trac.wordpress.org/changeset/33646/trunk/src/wp-includes/taxonomy.php you should then be safely able to delete the option to clear out the database row.

Hi @dd32, that fix worked for me but... how can I delete the hundreds and even thousands of cron tasks already created? will they be auto deleted sometime?

[boonebgorges]

Duplicate of #33423.

@Kent Brockman - Check out [33646], and in particular, the upgrade_431() routine. If you can't wait until 4.3.1, you can run a similar routine yourself (eg in an mu-plugin).

[leedo]

Replying to boonebgorges:

Duplicate of #33423.

@Kent Brockman - Check out [33646], and in particular, the upgrade_431() routine. If you can't wait until 4.3.1, you can run a similar routine yourself (eg in an mu-plugin).

We managed to end up with 150k of these cron events scheduled before rolling back to 4.2.4. I'm not sure if that is worse than expected, our setup is multisite and gets quite a bit of traffic. Running the code to unschedule the tasks in upgrade_431() takes prohibitively long with such a large list. I think there is probably a more efficient way, directly updating the entire option at once. Maybe most sites won't have 150k items to remove, but if they do, it will take much longer than can be run in a single HTTP request.

edit: Clearing it in one go like this was immediate, where as the above fix would probably take hours for us.

$cron = get_option("cron");
unset($cron["wp_batch_split_terms"]);
update_option("cron", $cron);

[boonebgorges]

Replying to leedo:

Replying to boonebgorges:

Duplicate of #33423.

@Kent Brockman - Check out [33646], and in particular, the upgrade_431() routine. If you can't wait until 4.3.1, you can run a similar routine yourself (eg in an mu-plugin).

We managed to end up with 150k of these cron events scheduled before rolling back to 4.2.4. I'm not sure if that is worse than expected, our setup is multisite and gets quite a bit of traffic. Running the code to unschedule the tasks in upgrade_431() takes prohibitively long with such a large list. I think there is probably a more efficient way, directly updating the entire option at once. Maybe most sites won't have 150k items to remove, but if they do, it will take much longer than can be run in a single HTTP request.

Yes, see the updated commits on #33423, which switch it to a single update_option() rather than a foreach loop. Also, note that this issue is not a symptom to trying to split all shared terms in a single HTTP request - WP tries to do it in batches. The issue in this ticket and #33423 is that in certain edge cases, the initial term-splitting cron job is not set during the normal 4.3 update routine.

[charlestonsw]

FWIW - the impact may be bigger than anticipated. This bug effectively disabled my site when the wp_option entry grew to over 16MiB.

MySQL disallows packets over that size by default which triggers all kinds of side effects; from 1MB log entries written to diskm to significant memory load on the web server, to DB connectivity issues.

Manually applied the patch and the @leedo manual update for the cron option. My site is no longer randomly dropping connections and average page load times for successful connections is back down to < 5s from what was > 25s prior to the patch.

May be prudent to escalate the 4.3.01 release? Not quite a security-level patch, but close IMO.

[David.Betts]

This bug has been killing me this last week. I kept migration my ec2 instances to bigger hardware and added a MySQL read-replica before this error started appearing:

2015/09/08 13:15:43 [error] 3545#0: *90 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 4194304 bytes) in /var/app/current/wp-includes/functions.php on line 384" while reading response header from upstream, client:

Hundreds of hours and thousands of dollars spent chasing this one.

[doublesharp]

I just had this bug kill my database server when it ran out of disk space. The constant updates to the cron option caused the binlog to grow at an exponential rate, using up 200GB in less than a week on a moderately trafficked site.

Lost a month of backups, hours of debugging, revenue from multiple sites being down. This could be an even bigger issue for database servers with replication.

I would also recommend prioritizing the release of a fix for this bug, even with the hotfix plugin available.

[knutsp]

Replying to doublesharp:

I would recommend a severity level of major/severe

This issue is already fixed and will be part of 4.3.1, coming soon. This ticket is therefore closed as a duplicate.

[doublesharp]

Replying to knutsp:

Replying to doublesharp:

I would recommend a severity level of major/severe

This issue is already fixed and will be part of 4.3.1, coming soon. This ticket is therefore closed as a duplicate.

Apologies, saw that just after I submitted my comment. Pretty tired from figuring out what was going on with my servers :/