Rollback Auto-Update (Rollback part 3)

[afragen]

This is Rollback part 3. It began with move_dir() in WP 6.2 for part 1. Part 2 was completed with #51857 in WP 6.3. This brings us to part 3.

Part 3 is Rollback for auto-updates. When manually updating plugins if the plugin has a fatal error on reactivation, the plugin is prevented from reactivating. Unfortunately, during an auto-update, this reactivation check doesn't occur and the the next time the site runs users will see the WSOD.

Rollback Auto-Update performs a similar re-activation check and if there is a fatal error it is captured in an error handler and the previously installed plugin is restored. If this occurs an email will be sent notifying the site admin of the failed update and rollback.

After the rollback, the pending auto-updating for core and theme updates are restarted.

This code is currently running for everyone who has the ​Rollback Update Failure feature plugin installed.

I personally have been testing this using a plugin that will fatal if the update occurs. The plugin is on my test site, active, and set to auto-update. I have been running like this since the beginning of the year.

The PR is slightly different than the code in the feature plugin. Please test, run the feature plugin on your site, and review the code in the PR. Mostly give us your comments and feedback.

props @costdev and @pbiron for continuing code review, rubber ducking, and sanity checks.

This PR adds the rollback auto update functionality to core. This is already part of the ​Rollback Update Failure feature plugin.

This requires ​PR2225.

Trac ticket: https://core.trac.wordpress.org/ticket/58281#ticket

[kirasong]

Thanks everyone for your work on this!

I left a couple of questions / comments in a review on the PR.

I'm going to try to do some further manual testing, but I don't currently see any blockers for getting a first run of this committed into trunk so that we can get wider testing.

[peterwilsoncc]

I've tested ​the linked pull request on our old friend Chassis running on VirtualBox and am seeing some apparent timeouts.

I wrote up a mini-plugin to ​increase the frequency the update cron jobs run so I didn't need to wait twelve hours for each round of testing. This allowed me to use wp-cron.php for events to emulate real world circumstances.

I then installed the following plugins:

• akismet 4.1.10
• gutenberg 16.5.1
• jetpack 11.3.2
• mailpoet 4.0.1
• soliloquy-lite 2.5.2.8
• woocommerce 7.0.0
• wordpress-seo 19.7
• wpforms-lite 1.7.0

Running with the pull request checked out, the updates were failing with the setup mentioned above. Aksimet, Gutenberg and Jetpack would update but the rest of them would fail to upgrade.

As the upgrade failed, no email notification was sent and the auto_updater.lock option would stay in the database preventing the next run from completing.

I am wondering if the PHP request is timing out. In wp-cron.php, one of the first things that runs is ​fast-finish code to prevent the requests from blocking sites. I am wondering if this is preventing the timeout from being extended when the updates begin running.

The biggest concern when plugins update automatically is that they may contain a fatal error that crashes a website.

This performs two loopback requests, one to the dashboard, and one to the homepage, in an attempt to detect such fatal errors.

Should a fatal error be detected, the update is restored to the previously installed version.

@costdev and I have been testing all combinations of plugins, fataling plugin, and themes with this, even @peterwilsoncc's list of plugins.

It tests out great!

Closing in favor of ​https://github.com/WordPress/wordpress-develop/pull/5287

[peterwilsoncc]

@costdev @afragen I tested the ​refreshed PR with the list of plugins I included in comment #14

With the plugins inactive, the updates ran successfully and I got the auto update email.

With the plugins active, it appears to have stalled after the WooCommerce update and I did not get the auto update email and the auto_updater.lock option remains. wordpress-seo and wpforms-lite were not updated.

With the plugins active I get 190ish queries on the plugins admin screen so it makes sense that being active may cause problems as the site is doing a lot more work than while they are inactive.

The backups of the successfully updated plugins remain in the upgrade-temp-backup/plugins folder.

As usual, this is testing on VirtualBox (using ​Chassis).

[afragen]

@peterwilsoncc I've run this using all of your plugins and several others, 14 in total including 2 plugins that will throw fatal errors.

My runs on LocalWP and the WP docker env have both been successful and the upgrade-temp-backups/plugins is empty.

Perhaps it's the method you use to help trigger the update? On LocalWP I'm able to use @costdev's cli-force-auto-updates.php plugin. On Docker I use npm run env:cli cron event run wp_version_check.

[costdev]

With additional testing, issues were reproduced.

Issues occur when browsing the site during an auto-update, and investigation is ongoing to find a solution. We'll continue work and try again for WordPress 6.5.

Thanks to all for your work and interest in Rollback Auto Update so far!

[kirasong]

Thanks so much @costdev @afragen @peterwilsoncc and everyone who has been working on and testing this feature.

I agree that given the regression in updates (updates not happening that would have succeeded) with the feature applied, and Beta 1 today/tomorrow, it makes sense to punt this to 6.5.

I'm planning to continue to follow this, and happy to help out in 6.5!

[annezazu]

Checking in here as we're less than a month out from beta 1. Is this still slated for 6.5? I want to ensure it's covered for testing if so.

[costdev]

@annezazu Thanks for checking in!

Yes the feature is still slated for 6.5. Unfortunately, I've been snowed under with agency work and haven't had time to circle back. In the Upgrade/Install Slack channel, ​I've reached out to other committers to see if they might have bandwidth to take a look at ​the PR.

With eyes and possible input from more committers, this feature can absolutely land in 6.5.

[swissspidy]

There hasn't been any traction here recently and beta 1 is only a week away. Do you perhaps want to raise this in dev chat, maybe with some testing instructions?

[costdev]

@swissspidy Thanks for flagging the lack of updates posted on the ticket!

There were additional rounds of testing done after I reached out in Slack. @peterwilsoncc will also be doing additional testing today. Let's wait and see if Peter finds any issues and see how this feature looks for getting into Beta 1.

[costdev]

Testing instructions can be found ​here.

[swissspidy]

No traction here since testing instructions were added.

Let's ensure we get this in early in 6.6 instead!

[audrasjb]

Thanks for the great work on this feature!
I added a few comments in the PR :)

[audrasjb]

Thanks for the detailed testing infos!

I tested the PR with the this-plugin-should-not-be-used plugin and the rollback works as expected and the upgrade-temp-backup is empty at the end of the process.

I also tested the PR with a plugin located on custom repositories (with or without a fatal error in the new version) and it looks like it works fine as well.

I added some minor code review comments, but in general it's looking good. I'm going to functionally test this over the next few days.

What shall we do with all the error_log() calls in this PR? Strip them out?

For debugging purposes, since this is a background task, it would be great if we could put them behind a WP_DEBUG flag so any reported issues can detail where things went wrong, then remove this logging later in the cycle.

I say leave them for now for broader testing. Somewhere around beta1 we can put the loopback results behind WP_DEBUG and remove most of the others.

[johnbillion]

In 58128:

Upgrade/Install: Automatically roll back to the previous version when an automatic plugin update results in a fatal error on the front end of the site.

This builds on the temporary backup system introduced in 6.3 to allow automatic updates to benefit from fatal error protection. A loopback request is performed to the home page of the site and the plugin is rolled back to its backed up version if a fatal error is observed.

For debugging and observability during beta, this change includes several calls to error_log() during the upgrade and rollback stages. These calls can be removed or placed behind a flag once we're ready for RC1.

Props costdev, johnbillion, mukesh27, afragen, audrasjb, justlevine, kirasong, peterwilsoncc

Fixes #58281

https://core.trac.wordpress.org/changeset/58128 🚀

[swissspidy]

Just as an FYI, these error_log calls are causing some WP-CLI tests to fail as there is now a lot of unexpected output when running commands.

In r58128, error_log() calls were added. One of these is intended to persist beyond release, the rest are for testing purposes during the 6.6 development cycle.

The testing error_log() calls caused ​test failures due to unexpected output.

This change guards the testing calls and the post-release call with WP_DEBUG and WP_DEBUG_LOG, similar to various block files' use of WP_DEBUG && WP_DEBUG_DISPLAY guards.

Looks good

[johnbillion]

In 58139:

Bootstrap/Load: Place the debugging output for automatic plugin and theme updates behind a debugging flag.

This change means this debugging output will only be shown when both the WP_DEBUG and WP_DEBUG_LOG constants are defined as true.

Props costdev, afragen, swissspidy

Fixes #58281

https://core.trac.wordpress.org/changeset/58139

[costdev]

@johnbillion Looks like we had some inconsistencies in the $is_debug definitions.

​PR 6632 should resolve the inconsistencies so that all $is_debug definitions used WP_DEBUG && WP_DEBUG_LOG.

@swissspidy Noticed that we had some inconsistencies in the debug guards and some of them checked WP_DEBUG_DISPLAY instead of WP_DEBUG_LOG.

Can you confirm that correcting the code to use WP_DEBUG && WP_DEBUG_LOG for all of the instances (which this PR does) will still prevent unexpected output in WP-CLI tests?

Thanks!

I think so, yes, When I check out this PR & run wp cron event run wp_version_check wp_update_plugins (which triggered these issues the last time) 👍

[costdev]

In 58308:

Upgrade/Install: Don't toggle maintenance mode on translation updates.

The Rollback Auto-Update feature introduced additional maintenance mode toggling.

After installing WordPress in a non-English (US) language, translation updates are performed automatically. As there may be a large number of updates for Core and bundled themes, users will be presented with a maintenance notice upon visiting the newly installed website.

To avoid concerning users that the website has failed to install correctly, this excludes translation updates from triggering the additional maintenance mode toggling.

Follow-up to [58128].

Props benniledl, afragen, rajinsharwar, costdev.
Fixes #61260. See #58281.

[costdev]

​PR 6632 has been reviewed by another component maintainer and has been tested by another committer.

Ready for commit. Note: @rogermedia revealed this issue during testing in the hosting Slack channel, and will be included in the props list.

[costdev]

In 58309:

Upgrade/Install: Make $is_debug consistent in WP_Automatic_Updater.

[58139] introduced debugging flags to ensure debugging output would only be shown when both the WP_DEBUG and WP_DEBUG_LOG constants are defined as true. However, some of the flags incorrectly use WP_DEBUG_DISPLAY rather than WP_DEBUG_LOG.

This fixes the flags to consistently use WP_DEBUG and WP_DEBUG_LOG as intended.

Follow-up to [58128], [58139].

Props rogermedia, afragen, swissspidy, costdev.
Fixes #58281.

[costdev]

In 58435:

Upgrade/Install: Delay automatic updates after installation.

After installation, the user is directed to the Log In page. This triggers the wp_schedule_update_checks() function which is hooked to init and schedules updates to run immediately if no other events exist. As a result of more robust use of maintenance mode for automatic updates added in [58128], the user may be presented with a maintenance mode screen just after installing WordPress.

To improve the user experience, this schedules core updates for 1 hour, plugin updates for 1.5 hours, and theme updates for 2 hours after installation.

Follow-up to [58128], [58139], [58308], [58309].

Props afragen, hellofromTonya, peterwilsoncc, nithi22, dd32.
Fixes #61457. See #58281, #61391.

[costdev]

In 58436:

Upgrade/Install: Disable maintenance mode when core auto-update fails.

In [58128], additional maintenance mode calls were added to the automatic updates process. However, there is an early return if a 'core' automatic update fails.

Maintenance mode isn't disabled until later in the WP_Automatic_Updater::update() method. This means that maintenance mode may continue to be enabled despite the core update being treated as a skipped update.

This disables maintenance mode before the early return.

Follow-up to [58128].

Props costdev, hellofromTonya, peterwilsoncc.
Fixes #61459. See #58281.

Thanks for the PR! Looks like this was merged in r58309.