Make WordPress Core

Opened 11 years ago

Closed 10 years ago

#26173 closed defect (bug) (fixed)

DB Upgrade Loop after Automatic Upgrade with db_version change and external object cache

Reported by: kirasong's profile kirasong Owned by: dd32's profile dd32
Milestone: 3.7.2 Priority: normal
Severity: major Version: 3.7.1
Component: Upgrade/Install Keywords: has-patch fixed-major
Focuses: Cc:


When running an automatic upgrade between db_version changes, it appears that the external object cache does not have its db_version updated, causing a db upgrade loop.

This is not an issue if a manual upgrade is performed via the admin screens.

To reproduce:

  • Install WordPress 3.7.1 with object-cache.php drop-in for your external object cache (Memcache & Memcached tested)
  • Enable Major Version upgrades (define( 'WP_AUTO_UPDATE_CORE', true );)
  • Install Beta Tester Plugin, and enable nightlies
  • Wait for schedule, or force upgrade with wp_version_check(); wp_maybe_auto_update();
  • Visit admin panel, and notice that you now have a loop for db upgrade: the database has the new db_version, and your object cache has the old one.

Chatted with @dd32 about this, and it seems this problem also exists with APC.

He noted he'd look into it further and attempt to reproduce again, but creating this bug to track efforts.
I'll update with a patch if I manage to track down a root cause.

Attachments (1)

26173.diff (560 bytes) - added by dd32 11 years ago.

Download all attachments as: .zip

Change History (13)

#1 @Ipstenu
11 years ago

  • Cc ipstenu@… added

#2 @knutsp
11 years ago

I had this issue on IIS7 with Wincache.

#3 @dd32
11 years ago

Finally tracked this down.. Bit of a hairy one.. so tread carefully, I might confuse. (for simplicity, db_version = 1 for old, db_version = 2 for new version)

  • The call to update the DB is normally done through a wp_remote_post( wp-admin/upgrade.php ) request from the upgrading script.
  • On the call to wp-admin/upgrade.php which updates the database, WP_INSTALLING is defined, this causes update_option(db_version, 2) to update the DB only, but not the caches. Once it's finished updating the DB, it calls wp_flush_cache() to clear the object cache.
  • In normal circumstances, ie. a Standard manual upgrade, the next pageload will cause the cache to be re-filled from the DB.
  • In the case of a Background Update however, after the DB upgrade step, we call update_option() while WP_INSTALLING is NOT defined, this causes us to update the object cache.

The problem is, that we store all options in one bucket, $prefix:options:alloptions which holds an array of every option in existence.

At this point, we've got the old data loaded in the local cache (db_version = 1), we update another entry in the array ( auto_core_update_notified ) and re-set it in the object cache (local & remote).
So we're overwriting the data in the cache with old data. The solution appears to be simple, call wp_cache_flush() after the call to wp-admin/upgrade.php takes place.

The root cause is much deeper, and is a bug in the way which we cache options. We prioritize the $alloptions cache above individual caches (which are only used for non-autoloaded values normally) which means reproducing this is very easy: Simply have two scripts running at the same time, and update two different options (both of which are autoloaded normally, or, one which is a new option) and one will overwrite the other in the cache.

Will attach a patch shortly.. I can think of ways to bypass it and cause the problem again.

Last edited 11 years ago by dd32 (previous) (diff)

11 years ago

#4 @dd32
11 years ago

  • Keywords has-patch added; needs-patch removed
  • Milestone changed from Awaiting Review to 3.7.2

#5 @dd32
11 years ago

  • Owner set to dd32
  • Resolution set to fixed
  • Status changed from new to closed

In 26448:

Core Updates: Fix a case where options (db_version specifically) can end up with stale values in the cache after a update is performed. Fixes #26173 for trunk.

#6 @dd32
11 years ago

  • Keywords fixed-major added
  • Resolution fixed deleted
  • Status changed from closed to reopened

#7 @nacin
11 years ago

So I'm caching backends, if I recall correctly, do not flush under certain situations. Should we be explicit about deleting db_version and alloptions here?

#8 @dd32
11 years ago

  • Keywords fixed-major removed

Hm, You do appear to be right.. Looking at Memcached for example:

function flush() {
	// Don't flush if multi-blog.
	if ( function_exists('is_site_admin') || defined('CUSTOM_USER_TABLE') && defined('CUSTOM_USER_META_TABLE') )
		return true;

Since all we need to do is to clear the alloptions cache from local memory, we could use wp_cache_delete( 'alloptions', 'options' ); here instead explicitly, the important part is that particular cache needs to go.

#9 @nacin
11 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 26734:

Core updates: Explicitly call wp_cache_delete() after a DB upgrade, as not all cache backends allow the entire backend to be flushed.

props dd32.
fixes #26173.

#10 @dd32
11 years ago

  • Keywords fixed-major added

This is currently on a 3.7.2 milestone, and was only fixed in trunk (3.8). Marking as such. We may want to shift to 3.8 if there's no 3.7.2.

#11 @nacin
10 years ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

#12 @nacin
10 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 27885:

Core Updates: Fix a case where options (db_version specifically) can end up with stale values in the cache after a update is performed.

Merges [26448] and [26734] from 3.8 to the 3.7 branch.

props dd32.
fixes #26173.

Note: See TracTickets for help on using tickets.