Make WordPress Core

Opened 8 years ago

Last modified 4 years ago

#37301 new feature request

A way to test plugin/theme installs and updates before applying

Reported by: zuige's profile Zuige Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version: 4.4.3
Component: Plugins Keywords: dev-feedback reporter-feedback ui-feedback ux-feedback
Focuses: ui, administration Cc:

Description

The # 1 reason for WordPress sites going completely down or becoming partially broken is the bad install or an update of a plugin or a theme.

The worst case scenario being a complete white screen of death from which the only way to recover is to manually remove the culprit plugin files, which unfortunately requires remote access to the server filesystem.

The current plugin upgrader checks if the upgrade causes errors and can in some cases recover from the most obvious cases syntax errors in the critical plugin code. This however fails to mitigate the slightly more subtle issues like incompatibilities between plugins/themes and javascript errors that may still completely wreck the site.

Furthermore, the more seasoned WordPress users are known to avoid updating plugins and themes on their site due to the volatility of running an update in production. This of course leaves their sites vulnerable to possible security issues.

The best practice of running staging environments to test plugins before updating requires a significant amount of technical knowledge and labour, and is not widely practiced among regular users.

I suggest we give the option for users to test plugin / theme installs and updates prior to applying them.

This requires a sort of sandbox environment for the user where they can freely install and test plugins without worrying about breaking anything.

I've created an example implementation of these plugin update sandboxes in (slightly hacky) plugin form. The plugin adds a "test update" button to the update.php admin screen, which creates a sandbox where the user can test the updated plugin without affecting the live site. Both the plugin directory and the database are separated from the live ones.

https://github.com/anttiviljami/wp-safe-updates

Screenshots:

The "test update" button in update.php

https://github.com/anttiviljami/wp-safe-updates/raw/master/assets/screenshot-1.png

Updating the plugin in an alternate heap (sandbox)

https://github.com/anttiviljami/wp-safe-updates/raw/master/assets/screenshot-2.png

Testing the plugin inside the sandbox

https://github.com/anttiviljami/wp-safe-updates/raw/master/assets/screenshot-3.png

--

I would love more discussion about this. I feel the safety of theme / plugin updates is definitely among the most important issues to solve for the future of WordPress security.

Love,
@anttiviljami

Attachments (1)

wp-safe-updates.zip (386.7 KB) - added by Zuige 8 years ago.
WP Safe Updates plugin

Download all attachments as: .zip

Change History (18)

@Zuige
8 years ago

WP Safe Updates plugin

#1 @swissspidy
8 years ago

Related: #31532, #31744

There's also some work going on for Shiny Updates V3:

https://github.com/obenland/shiny-updates/issues/288
https://github.com/obenland/shiny-updates/issues/292

I like that comment from the first ticket:

We also need to take care of updates and not allowing updates to a version that contains a fatal error. I would love for us to store the current version and restore that version if an update fails, but that may be out of scope for now.

As pointed out, WordPress currently has kind of a plugin sandbox so it can notify users if a plugin activation failed. That's what I love about it — it just works.

I agree that staging environments are considered a best practice, but the average user probably has no idea what that is. However, I don't think it's suitable for core to give the user an option to test drive this in a semi-separate environment, “where they can freely install and test plugins without worrying about breaking anything”.That sounds like a big attack surface and something not many users would use or understand. I'm pretty sure that symlinks won't work on many hosts either. Hence I consider this being plugin material, so thanks for creating this plugin.

For core, I'd rather have automatic tests of updates and rollbacks if there's a fatal error.

#2 @Zuige
8 years ago

@swissspidy Thanks for your comments!

You seem to be mostly referring to the capability of checking for the most obvious issues like fatal php errors on the site. As mentioned, this already kind of exists but clearly isn't a sufficient solution.

What about javascript errors and incompatibilities?
What if the update completely changes how my theme looks, but the author didn't tell me that beforehand?
What if another plugin depended on an API provided by the old version of the plugin, but that went missing in the update?

However, I don't think it's suitable for core to give the user an option to test drive this in a semi-separate environment, “where they can freely install and test plugins without worrying about breaking anything”.That sounds like a big attack surface and something not many users would use or understand.

You're right. The security aspects of such a feature have to be considered. This is not a small feature that can just be slapped on top of core and hope it works for most people. The feature needs a lot of careful thought about how to implement it in a safe manner.

As for the not understading part, some good hosting providers already do provide an on-demand staging environment sandbox feature, similar to what the plugin does. In terms of UX, having a "sandbox for tests" is a fairly easy thing to understand even for the less technical user. Especially after something's gone wrong in a previous update.

I definitely agree that the way forward would be automatic tests for updates. However, proper visual regression / functionality acceptance testing (done in a headless browser) is hard and still requires a sandboxed test environment. Simply running a bunch of unit tests doesn't actually tell you if the site works the way the user expects.

Props for pointing out the missing rollback feature. Is there a separate ticket for that in trac already? There should definitely be one.

Last edited 8 years ago by Zuige (previous) (diff)

#3 @Zuige
8 years ago

I think the core question for this issue is, whether it's worth the significant effort for WordPress to implement a sandbox type feature to pursue a safer method for plugin updates and installs.

We already have theme previews in the Customizer, which is a kind of sandbox. Why not have plugin previews, and previews for updates as well?

#4 @swissspidy
8 years ago

As for the not understanding part, some good hosting providers already do provide an on-demand staging environment sandbox feature, similar to what the plugin does.

That's why I think this should be better handled by hosts or plugins. I suggest proposing your plugin as a feature project to gather some feedback and continue work on it in a group. It's usually the best way to work on something big like this.

Props for pointing out the missing rollback feature. Is there a separate ticket for that in trac already? There should definitely be one.

Yeah, #31532 is the ticket for that. I'm looking into it as part of the Shiny Updates feature plugin.

#5 @Zuige
8 years ago

I'll look into the feature plugin project idea. Thanks!

Yeah, #31532 is the ticket for that. I'm looking into it as part of the Shiny Updates feature plugin.

I meant rollback as a general, user invokable feature. Not just automatic rollback for when an error is detected.

Most often you won't notice right away if something breaks after the update.

#6 follow-ups: @jdgrimes
8 years ago

I have been thinking about something like this for a while now. What I was thinking though was that rather than the user manually testing, each plugin would just include a suite of tests that WordPress would run automatically in the background each time it detected a new update. So the update would be hidden if it wasn't compatible, or the user would be given a warning message if they attempted to update. If the tests passed, the user would just see the update as they normally do and just has to click update to install.

However, until plugins adopted this and started shipping automated tests with the source, manually checking by the user would still be necessary, I guess.

If you start working on a feature project @Zuige, let me know. I'd be interested in participating. I think automatic update verification is the future of the web.

#7 in reply to: ↑ 6 @Zuige
8 years ago

[double post deleted]

Last edited 8 years ago by Zuige (previous) (diff)

#8 in reply to: ↑ 6 @Zuige
8 years ago

Replying to jdgrimes:

I have been thinking about something like this for a while now. What I was thinking though was that rather than the user manually testing, each plugin would just include a suite of tests that WordPress would run automatically in the background each time it detected a new update. So the update would be hidden if it wasn't compatible, or the user would be given a warning message if they attempted to update. If the tests passed, the user would just see the update as they normally do and just has to click update to install.

However, until plugins adopted this and started shipping automated tests with the source, manually checking by the user would still be necessary, I guess.

My company https://seravo.com has been doing automated update testing for up to thousands of wordpress sites for the last 2 years very succesfully. Automated updates are definitely the way of the future. The key to making them possible is using sandboxed environments, which only the best hosting providers have. Even less have testing tools like PhantomJS or testing suites available.

I would like to bring a sandboxing feature to WordPress core, so we could have non-hosting providers do the automated testing side of things as a service for even the cheapest hosts (on which still the majority of WordPress sites run). Acceptance update testing for WordPress could well be the next big business opportunity.

Even as is, the sandbox feature shown possible in the WP Safe Updates plugin would be immensely useful for those sites that don't have access to on-demand staging environments.

If you start working on a feature project @Zuige, let me know. I'd be interested in participating. I think automatic update verification is the future of the web.

Please do participate. The development of this feature plugin is and always will be open source. Your contribution would be more than welcome!

#9 @Zuige
8 years ago

Update: The Safe Updates plugin is now available on the plugin directory.

https://wordpress.org/plugins/wp-safe-updates/

This ticket was mentioned in Slack in #feature-shinyupdates by anttiviljami. View the logs.


8 years ago

#11 @Zuige
8 years ago

  • Keywords dev-feedback reporter-feedback ui-feedback ux-feedback added

This ticket was mentioned in Slack in #core by anttiviljami. View the logs.


8 years ago

This ticket was mentioned in Slack in #design by karmatosed. View the logs.


6 years ago

This ticket was mentioned in Slack in #core-php by afragen. View the logs.


6 years ago

This ticket was mentioned in Slack in #design by karmatosed. View the logs.


5 years ago

#16 @boemedia
5 years ago

In the current WordPress version (5.2), there’s the new core functionality of Site Health Check and Recovery mode / White Screen Of Death protection. However, these are fixes to help you access your site after things went wrong and find the culprit.

I do see a benefit of testing before fixing, and I wonder if this fuction can be integratied in Site Health Check by adding this test feature and make sure plugins / themes are working properly.

This ticket was mentioned in Slack in #core by timothybjacobs. View the logs.


4 years ago

Note: See TracTickets for help on using tickets.