DB Upgrade Loop after Automatic Upgrade with db_version change and external object cache

[kirasong]

When running an automatic upgrade between db_version changes, it appears that the external object cache does not have its db_version updated, causing a db upgrade loop.

This is not an issue if a manual upgrade is performed via the admin screens.

To reproduce:

• Install WordPress 3.7.1 with object-cache.php drop-in for your external object cache (​Memcache & ​Memcached tested)
• Enable Major Version upgrades (define( 'WP_AUTO_UPDATE_CORE', true );)
• Install ​Beta Tester Plugin, and enable nightlies
• Wait for schedule, or force upgrade with wp_version_check(); wp_maybe_auto_update();
• Visit admin panel, and notice that you now have a loop for db upgrade: the database has the new db_version, and your object cache has the old one.

​Chatted with @dd32 about this, and it seems this problem also exists with APC.

He noted he'd look into it further and attempt to reproduce again, but creating this bug to track efforts.
I'll update with a patch if I manage to track down a root cause.

[knutsp]

I had this issue on IIS7 with Wincache.

[dd32]

Finally tracked this down.. Bit of a hairy one.. so tread carefully, I might confuse. (for simplicity, db_version = 1 for old, db_version = 2 for new version)

• The call to update the DB is normally done through a wp_remote_post( wp-admin/upgrade.php ) request from the upgrading script.
• On the call to wp-admin/upgrade.php which updates the database, WP_INSTALLING is defined, this causes update_option(db_version, 2) to update the DB only, but not the caches. Once it's finished updating the DB, it calls wp_flush_cache() to clear the object cache.
• In normal circumstances, ie. a Standard manual upgrade, the next pageload will cause the cache to be re-filled from the DB.
• In the case of a Background Update however, after the DB upgrade step, we call update_option() while WP_INSTALLING is NOT defined, this causes us to update the object cache.

The problem is, that we store all options in one bucket, $prefix:options:alloptions which holds an array of every option in existence.

At this point, we've got the old data loaded in the local cache (db_version = 1), we update another entry in the array ( auto_core_update_notified ) and re-set it in the object cache (local & remote).
So we're overwriting the data in the cache with old data. The solution appears to be simple, call wp_cache_flush() after the call to wp-admin/upgrade.php takes place.

The root cause is much deeper, and is a bug in the way which we cache options. We prioritize the $alloptions cache above individual caches (which are only used for non-autoloaded values normally) which means reproducing this is very easy: Simply have two scripts running at the same time, and update two different options (both of which are autoloaded normally, or, one which is a new option) and one will overwrite the other in the cache.

Will attach a patch shortly.. I can think of ways to bypass it and cause the problem again.

[dd32]

In 26448:

Core Updates: Fix a case where options (db_version specifically) can end up with stale values in the cache after a update is performed. Fixes #26173 for trunk.

[nacin]

So I'm caching backends, if I recall correctly, do not flush under certain situations. Should we be explicit about deleting db_version and alloptions here?

[dd32]

Hm, You do appear to be right.. Looking at Memcached for example:

function flush() {
	// Don't flush if multi-blog.
	if ( function_exists('is_site_admin') || defined('CUSTOM_USER_TABLE') && defined('CUSTOM_USER_META_TABLE') )
		return true;

Since all we need to do is to clear the alloptions cache from local memory, we could use wp_cache_delete( 'alloptions', 'options' ); here instead explicitly, the important part is that particular cache needs to go.

[nacin]

In 26734:

Core updates: Explicitly call wp_cache_delete() after a DB upgrade, as not all cache backends allow the entire backend to be flushed.

props dd32.
fixes #26173.

[dd32]

This is currently on a 3.7.2 milestone, and was only fixed in trunk (3.8). Marking as such. We may want to shift to 3.8 if there's no 3.7.2.

[nacin]

In 27885:

Core Updates: Fix a case where options (db_version specifically) can end up with stale values in the cache after a update is performed.

Merges [26448] and [26734] from 3.8 to the 3.7 branch.

props dd32.
fixes #26173.