WordPress.org

Make WordPress Core

Opened 21 months ago

Closed 21 months ago

Last modified 21 months ago

#33463 closed defect (bug) (duplicate)

Flooded 'wp_batch_split_terms' in cron

Reported by: array064 Owned by:
Milestone: Priority: normal
Severity: normal Version: 4.3
Component: General Keywords:
Focuses: Cc:

Description

After I upgrade Wordpress, from version 4.2.4 to version 4.3, there was an oddity: in the table PREFIX_options with option_name = 'cron', on option_value contains a lot of cron associated with 'wp_batch_split_terms' (see attachment I sent). This makes our site to be very very slow, especially on the back-end (wp-admin).

Attachments (1)

j9i9bjd_options - cron.sql (3.0 MB) - added by array064 21 months ago.

Change History (13)

#1 @dd32
21 months ago

  • Milestone Awaiting Review deleted
  • Resolution set to duplicate
  • Status changed from new to closed

Hi @array064 Welcome to Trac :)

This is being tracked in #33423 and was fixed in [33646]

#2 @array064
21 months ago

Thank you for your quick response. :) So, while I am waiting for Wordpress version 4.3.1 which contains the fix, what should I do? Because cron in the table PREFIX_options progressively enlarged with the 'wp_batch_split_terms'.

Last edited 21 months ago by array064 (previous) (diff)

#3 follow-up: @dd32
21 months ago

@array064 You should fix it by apply this change to prevent it filling up https://core.trac.wordpress.org/changeset/33646/trunk/src/wp-includes/taxonomy.php you should then be safely able to delete the option to clear out the database row.

#4 in reply to: ↑ 3 @Kent Brockman
21 months ago

  • Resolution duplicate deleted
  • Status changed from closed to reopened

Replying to dd32:

@array064 You should fix it by apply this change to prevent it filling up https://core.trac.wordpress.org/changeset/33646/trunk/src/wp-includes/taxonomy.php you should then be safely able to delete the option to clear out the database row.

Hi @dd32, that fix worked for me but... how can I delete the hundreds and even thousands of cron tasks already created? will they be auto deleted sometime?

#5 follow-up: @boonebgorges
21 months ago

  • Resolution set to duplicate
  • Status changed from reopened to closed

Duplicate of #33423.

@Kent Brockman - Check out [33646], and in particular, the upgrade_431() routine. If you can't wait until 4.3.1, you can run a similar routine yourself (eg in an mu-plugin).

#6 in reply to: ↑ 5 ; follow-up: @leedo
21 months ago

Replying to boonebgorges:

Duplicate of #33423.

@Kent Brockman - Check out [33646], and in particular, the upgrade_431() routine. If you can't wait until 4.3.1, you can run a similar routine yourself (eg in an mu-plugin).

We managed to end up with 150k of these cron events scheduled before rolling back to 4.2.4. I'm not sure if that is worse than expected, our setup is multisite and gets quite a bit of traffic. Running the code to unschedule the tasks in upgrade_431() takes prohibitively long with such a large list. I think there is probably a more efficient way, directly updating the entire option at once. Maybe most sites won't have 150k items to remove, but if they do, it will take much longer than can be run in a single HTTP request.

edit: Clearing it in one go like this was immediate, where as the above fix would probably take hours for us.

$cron = get_option("cron");
unset($cron["wp_batch_split_terms"]);
update_option("cron", $cron);
Last edited 21 months ago by leedo (previous) (diff)

#7 in reply to: ↑ 6 @boonebgorges
21 months ago

Replying to leedo:

Replying to boonebgorges:

Duplicate of #33423.

@Kent Brockman - Check out [33646], and in particular, the upgrade_431() routine. If you can't wait until 4.3.1, you can run a similar routine yourself (eg in an mu-plugin).

We managed to end up with 150k of these cron events scheduled before rolling back to 4.2.4. I'm not sure if that is worse than expected, our setup is multisite and gets quite a bit of traffic. Running the code to unschedule the tasks in upgrade_431() takes prohibitively long with such a large list. I think there is probably a more efficient way, directly updating the entire option at once. Maybe most sites won't have 150k items to remove, but if they do, it will take much longer than can be run in a single HTTP request.

Yes, see the updated commits on #33423, which switch it to a single update_option() rather than a foreach loop. Also, note that this issue is not a symptom to trying to split all shared terms in a single HTTP request - WP tries to do it in batches. The issue in this ticket and #33423 is that in certain edge cases, the initial term-splitting cron job is not set during the normal 4.3 update routine.

#8 @charlestonsw
21 months ago

FWIW - the impact may be bigger than anticipated. This bug effectively disabled my site when the wp_option entry grew to over 16MiB.

MySQL disallows packets over that size by default which triggers all kinds of side effects; from 1MB log entries written to diskm to significant memory load on the web server, to DB connectivity issues.

Manually applied the patch and the @leedo manual update for the cron option. My site is no longer randomly dropping connections and average page load times for successful connections is back down to < 5s from what was > 25s prior to the patch.

May be prudent to escalate the 4.3.01 release? Not quite a security-level patch, but close IMO.

#9 @David.Betts
21 months ago

This bug has been killing me this last week. I kept migration my ec2 instances to bigger hardware and added a MySQL read-replica before this error started appearing:

2015/09/08 13:15:43 [error] 3545#0: *90 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 4194304 bytes) in /var/app/current/wp-includes/functions.php on line 384" while reading response header from upstream, client:

Hundreds of hours and thousands of dollars spent chasing this one.

#10 follow-up: @doublesharp
21 months ago

  • Severity changed from normal to major

I just had this bug kill my database server when it ran out of disk space. The constant updates to the cron option caused the binlog to grow at an exponential rate, using up 200GB in less than a week on a moderately trafficked site.

Lost a month of backups, hours of debugging, revenue from multiple sites being down. This could be an even bigger issue for database servers with replication.

I would also recommend prioritizing the release of a fix for this bug, even with the hotfix plugin available.

Last edited 21 months ago by doublesharp (previous) (diff)

#11 in reply to: ↑ 10 ; follow-up: @knutsp
21 months ago

  • Severity changed from major to normal

Replying to doublesharp:

I would recommend a severity level of major/severe

This issue is already fixed and will be part of 4.3.1, coming soon. This ticket is therefore closed as a duplicate.

#12 in reply to: ↑ 11 @doublesharp
21 months ago

Replying to knutsp:

Replying to doublesharp:

I would recommend a severity level of major/severe

This issue is already fixed and will be part of 4.3.1, coming soon. This ticket is therefore closed as a duplicate.

Apologies, saw that just after I submitted my comment. Pretty tired from figuring out what was going on with my servers :/

Note: See TracTickets for help on using tickets.