Make WordPress Core

Opened 10 years ago

Closed 4 years ago

#27669 closed defect (bug) (invalid)

Stale `db_version` value after update with external object cache

Reported by: johnbillion's profile johnbillion Owned by:
Milestone: Priority: normal
Severity: critical Version: 3.9
Component: Upgrade/Install Keywords: has-patch needs-testing
Focuses: Cc:

Description

This is similar to #26173 but not quite the same.

My site runs trunk with the WordPress Beta Tester and Memcached plugins. The site just updated to [27916] and I'm getting the "No Update Required" loop which prevents any access to the admin area.

The db_version option in my database has been updated to the correct value (27916) but get_option('db_version') on any admin page returns a stale value (26691). Unlike #26173, the option has been updated with the correct value but the cache remains stale (instead of updating the option with a stale value).

On /wp-admin/upgrade.php the object cache is suspended so the non-stale value is returned, which results in the "No Update Required" message.

Attachments (3)

2015-05-07_143801.jpg (111.0 KB) - added by thburkart 9 years ago.
DATABASE UPDATE REQUIRED - LOOP
27669.diff (731 bytes) - added by kraftbj 9 years ago.
flush cache on multisite network upgrade db ver bump
27669.2.diff (3.2 KB) - added by Mike_Cowobo 8 years ago.
Forces db_version to be fetched from the db if it's different from $wp_db_version

Download all attachments as: .zip

Change History (28)

#1 @nacin
10 years ago

How does [27885] not clear the cache?

#2 @johnbillion
10 years ago

  • Milestone Awaiting Review deleted
  • Resolution set to invalid
  • Status changed from new to closed

This appears to be an issue local to the object cache on the server.

#3 @nacin
10 years ago

  • Milestone set to 3.9.1
  • Resolution invalid deleted
  • Status changed from closed to reopened
  • Summary changed from Stale `db_version` value after automatic update with external object cache to Stale `db_version` value after update with external object cache

Re-opening and generalizing to refer to any kind of update, not just automatic ones. Until we know for sure, I am inclined to think this is a race condition, and possibly one that is more likely to happen in an auto update. I seem to remember reports that this can happen with a manual update too.

Moving to 3.9.1 for investigation.

See also this conversation: https://twitter.com/Rarst/status/457275387727581184

This thread: http://wordpress.stackexchange.com/questions/141731/wordpress-3-9-database-update-infinite-loop?stw=2

And this thread: http://wordpress.org/support/topic/cant-access-dashboard-after-383-update?replies=6#post-5456633

#4 @nacin
10 years ago

While this is bad, at this time we won't be having a DB version bump in 3.9.1, which means minor release updates from 3.9 to 3.9.1 won't be affected. We have some time to properly investigate this.

Because we can patch wp-admin/includes/update-core.php (or wp-admin/upgrade.php — which would be copied over by the time it gets run), we can solve this in a point release that introduces a DB upgrade, rather than being forced to introduce it in one that precedes it.

#5 @nacin
10 years ago

  • Milestone changed from 3.9.1 to 3.9.2

#6 @helen
10 years ago

3.9.2 was shipped without this - what do we need to do here, if anything?

#7 @willmot
10 years ago

3.9.2 was shipped without this - what do we need to do here, if anything?

We haven't run into the issue this on any of the sites we updated.

#8 @helen
10 years ago

  • Milestone changed from 3.9.2 to Future Release

Still not sure what needs to be done here, so punting.

#9 @Rarst
9 years ago

I get this repeatedly on one of my sites (curiously only one among several sharing same hosting account and object cache setup).

As of right now (auto update to 4.2.1 got stuck my admin yet again, as did 4.2 before it) I have 31533 value of db_version in database and 31532 reported by wp-cli.

#10 @majemedia
9 years ago

I also ran into this last night. My fix was to add define('WP_CACHE_KEY_SALT', md5(DB_NAME . __FILE__)); to my wp-config.php on each site using that memcached server and then restart memcached.

#11 @Rarst
9 years ago

I am using SiteGround's plugin which does define salt for object cache https://plugins.trac.wordpress.org/browser/sg-cachepress/trunk/memcache.tpl#L16

Note that simply flushing cache in any way would make the issue go away... until next time.

#12 @rmccue
9 years ago

I suspect this could be #31245. @willmot mentioned we haven't hit it on Human Made stuff, and I'd guess that's because we're running a modified version of the Memcache plugin.

It's somewhat trivial to cause:

  • WP runs a HTTP request to itself to update the DB on autoupgrades.
  • A plugin adds a hook to shutdown or similar which calls update_option or similar
  • HTTP request closes (fastcgi_finish_request or similar in the shutdown hook), and WP flushes the cache
  • The race condition causes the plugin to call update_option, which overwrites alloptions

The key to causing it here is fastcgi_finish_request though, which causes the process to continue running after the request has finished. There's other ways to cause it though, things like TLC Transients can also cause it by spawning a grandchild process that does the same.

Occam's Razor indicates that this might not be the case, as it's somewhat complex to cause, but the fact we've not hit it on HM stuff, and that we already have wp_cache_delete( 'alloptions', 'options' ) to mitigate #31245 is evidence in favour of this theory.

#13 @Ipstenu
9 years ago

#26676 was marked as a duplicate.

This ticket was mentioned in Slack in #forums by ipstenu. View the logs.


9 years ago

This ticket was mentioned in Slack in #forums by ipstenu. View the logs.


9 years ago

@thburkart
9 years ago

DATABASE UPDATE REQUIRED - LOOP

#16 @SergeyBiryukov
9 years ago

Ran into this as well after both 4.2.1 and 4.2.2, using the Memcached plugin.

#17 @johnbillion
9 years ago

As Ryan pointed out, #31245 might be the culprit here. Leaving this open for now.

This ticket was mentioned in Slack in #core by johnbillion. View the logs.


9 years ago

#19 @kraftbj
9 years ago

Related: Multisite upgrade network notices can experience the stale cache. Applying the logic of [27885] in wp-admin/network/upgrade.php resolved the previously constant upgrade network notice for me when using https://wordpress.org/plugins/redis-cache/

@kraftbj
9 years ago

flush cache on multisite network upgrade db ver bump

This ticket was mentioned in Slack in #core by dd32. View the logs.


9 years ago

This ticket was mentioned in Slack in #core by kraft. View the logs.


8 years ago

#22 @johnbillion
8 years ago

  • Keywords has-patch needs-testing added

#23 @Mike_Cowobo
8 years ago

Another scenario that could trigger this is when WP is updated on in a staging environment and deployed without a cache flush. This should obviously not happen at all, but I saw this happen recently one of the bigger WP hosting providers offering integrated staging.

To reproduce the "No Update Required" loop with WP CLI:

$alloptions = wp_load_alloptions();
$alloptions['db_version'] = 36685; // Any version but $wp_db_version
wp_cache_set( 'alloptions', $alloptions, 'options' );

A more robust solution would be to actually check the option in the db in wp-admin/admin.php when db_version != $wp_db_version and flush the cache if a discrepancy is detected. This will:

  • Only cause a DB query when upgrade is required, or when something is actually wrong
  • Always work, regardless of when and where db_version was updated

@Mike_Cowobo
8 years ago

Forces db_version to be fetched from the db if it's different from $wp_db_version

This ticket was mentioned in Slack in #core by sergey. View the logs.


7 years ago

#25 @johnbillion
4 years ago

  • Milestone Future Release deleted
  • Resolution set to invalid
  • Status changed from reopened to closed

No movement in four years. Closing. Haven't seen those problem myself in a long time.

Note: See TracTickets for help on using tickets.