`update_option()` race condition with non-autoloaded options

[dd32]

Starting back in [25664] there's a race condition with object caches where get_option() will return a value, but update_option() will refuse to update it.
This is kind-of-related to alloptions, but affects non-autoloaded options (which are not in alloptions, but in their own cache).

Consider the following scenario:

Process 1: Fetch option_a. Fills local cache. 
Process 2: Update option_a.
Process 1: Delete option_a.
Process 2: Update DB.
Process 1: Delete from DB. Delete from remote cache.
Process 2: Update remote cache.
...
Process 5: Fetch option_a. Fills local cache from remote cache.
Process 5: Update option_a.  FAIL. DB doesn't have option. abort. cache not updated.
...
(repeat process 5 above many times)
Process 10: Get option_a (Still the value that Process 2 set in cache, even though it's not in the DB)
Process 10: Update option_a (Doing the same as Process 5 now). FAIL.

Seems very racey and unlikely, but I've seen it happen on WordPress.org at least twice in the last few weeks when we update Jetpack (which makes heavy usage of the options table, and is loaded on most WordPress.org requests).

When it happens, get_option() will continue to return a stale value from the cache that no longer exists in the DB and update_option() will fail to update the option as long as it exists in cache.
If a plugin is performing an operation to update the stale option often, it can cause a huge load spike on the database server of never-ending failing UPDATE queries.
The only way to 'fix' it is to create the DB row manually, or flush the object cache key.

The patch attached attempts to perform an add_option() in the case of the update_option() DB query failing.
This isn't exactly a new behaviour for options - add_option() will effectively perform an update_option() in the event the option you're trying to add already exists (through it using INSERT .. ON DUPLICATE KEY UPDATE..), doing it in reverse doesn't seem that out of the question.

[johnjamesjacoby]

Possibly related: #40052, r5248

[SergeyBiryukov]

Related: #51352, #51372.