Make WordPress Core

Opened 4 weeks ago

Last modified 7 days ago

#43395 new task (blessed)

Add Automated E2E Tests for Core Updates

Reported by: jorbin Owned by:
Milestone: WordPress.org Priority: normal
Severity: normal Version:
Component: Upgrade/Install Keywords:
Focuses: Cc:


This is a follow up to the issues introduced in #43103 and described in https://make.wordpress.org/core/2018/02/06/wordpress-4-9-4-release-the-technical-details/

Essentially, due to the way automated updates work, we need automated tests that can confirm they are working. Due to the code loading path differences between phpunit, wp-cli, wp-cron, and the WordPress UI, it's important that this test work as close as possible to the way the majority of sites are autoupdated.

Change History (12)

#1 @jorbin
4 weeks ago

  • Keywords 2nd-opinion added

My idea for the route to solving this is slightly complicated, but I think does the best job of ensuring that automatic updates work.

1) Create a docker container for running the tests. We may be able to use some prior art from Gutenberg. This should include file permissions where auto updates will work. We should install the latest stable version of wordpress (though it may make sense in the future to also do this on a branch by branch basis)
2) Include and activate the beta tester plugin. Configure it for minor version nightly updates.
3) Trigger an autoupdate via a hit to wp-cron.
4) Confirm that the site doesn’t fatal and the site is updated. Maybe also confirm that the error log is empty.

cc/ @dd32 and @pento to see if they agree.

#2 @pento
4 weeks ago

I've been chatting to @dd32 about this, I think we've all come to similar conclusions.

Docker is definitely the best option here, as we're going to need to test a bunch of variations of WordPress versions (both being upgraded from and to), PHP versions, file permissions, etc. Dion has a more comprehensive list of variations that he'll post to this ticket.

The first phase should probably just be something super simple. Use WP-CLI to install 4.9.0, then run wp_maybe_auto_update() to force the update synchronously.

This will give us something to base it on, we have quite a few Docker experts in the WordPress Core community who can contribute to expand this, too.

I hacked together a wordpress-develop Docker config a few weeks ago, some of this could be reused as the basis for this project, but I think this would be best done in a repo outside of wordpress-develop. We're building a tool that installs various versions of WordPress, wordpress-develop doesn't really have any relation to it.

As far as timing goes, I may have a little time in the coming weeks, though Gutenberg is still taking most of my time. If someone wants to run with this in the mean time, we can definitely get a wordpress-e2e repo going on the GitHub org for better collaboration.

This ticket was mentioned in Slack in #core by jeffpaul. View the logs.

3 weeks ago

#4 @dd32
3 weeks ago

After speaking with a few people, here's the list I've come up with of a MVP and also a huge stretch goal of various matrix options which we can/should explore.
The matrix below is huge and I'm sure I've missed a few angles, some things may not make sense to run together, but at the end of the day the WordPress upgrade routines run in a huge number of varied environments, and previously most of the testing has had to be done manually in the past due to lack of tooling available to automate it.

A lot of what is presented below is just a brain dump of the various branches of code we have, some will make sense to run a full upgrade with, some probably only need some basic unit testing (Variants of file system paths would benefit from a small handful of phpunit tests run against core in a docker instance for example, but some should maybe get a full upgrade run).


  • Running an update of trunk (HEAD/master) to the latest nightly via various trigger methods (See Various methods of updates below). Trust WordPress when it says it completed successfully.

More detailed ideas and stretch goals:

  • Running upgrades in Docker locally & via Travis CI seems to be the best method. Docker Compose & custom published DockerFiles should help there
  • Testing various types of upgrade
    • Minor x.y.Z => x.y.(Z+1) - With the expectation of it also failing under certain circumstances
    • Major x.Y => x.(Y+1)
    • Dev nightly (Trunk), Possibly Stable branch nightly while we're at it.
  • Various methods of updates:
    • Background Update (Triggered by wp_version_check() running in a cron environment)
    • Background Update (Triggered by calling wp_maybe_auto_update() - as some hosts do)
    • Via wp-cli (Depending on if it differs from the above methods?)
    • Manual updates through the UI - Clicking the button (Stretch goal of using a modern fake-browser)
  • Variants of filesystem method access
    • direct - expect that PHP owns and can modify all files - MVP
    • direct_gw - expect that PHP can modify own files, but not create them. PHP user is in the same group as the file owner. Should only be possible to do Minor updates
    • ftp - Have files be RO except via FTP.
      • Variant: FTP Extension
      • Variant: FTP "Sockets" (php ftp extension not available)
      • Use different FTP servers in different configurations. ProFTPD, PureFTPd and vsftpd - possibly one configured as a windows-ftp server (Yes, some Windows FTP servers have a different ls output)
  • Variants of file system paths
    • Nested installs
    • .svn/.git directories at various levels to block updates (We can probably relax that though)
    • PHP __DIR__ not matching FTP paths
      • FTP may be chroot()'d presenting /users/dd32/domain/xyz.com as /domain/xyz.com
      • May rewrite paths completely, presenting /home/dd32/domain/xyz.com as /users/1234/xyz.com
  • Verify that WordPress actually upgraded successfully, and not just that WordPress said it completed successfully.
  • Verify WordPress respects the various update filters and constants for how it updates

Random ideas:

  • Dockerise the WordPress.org API and/or use a mu-plugin to force WordPress to see the API response we want it to (So it doesn't hit the WodPress.org api at all). That would allow for easier testing of major/minor updates.
  • Fetch zips locally that we create, to allow us to create a "fake" release zip. For example, current stable is WordPress 5.0.0, we create a WordPress 5.5.0 zip as a manner to test that major updates can succeed from trunk to next-trunk.

I don't have my full notes with me right now, but if I've missed anything major i'll update this later in the week when I'm back in front of them.
I hope this is useful to someone else as well to kind of give some direction on what kinds of things we need to test here.

I've personally run into a few brick walls while trying to get a basic MVP of this working, due to my limited experience with docker and related tooling, there's a great amount of iterative work which could be done once a MVP is running though. While I'll personally keep working on attempting this when I can, if there's anyone else who has the available time and knowledge to scaffold something basic up, that'd be a great start for the community.

This ticket was mentioned in Slack in #core by audrasjb. View the logs.

3 weeks ago

#6 @voldemortensen
2 weeks ago

I made a thing.

Currently, it just runs through every version back to 3.7 and runs wp core update <url to update zip>, but it would be easy to expand on this. Of course, run.pl will need to be converted to PHP, Perl is just my go to language.

In run.pl, instead of running wp core update <url>, we could run wp eval '//update code' or any sort of custom thing really.

The first run takes awhile as it has to build custom images, but once those are cached it moves pretty quickly.

This ticket was mentioned in Slack in #core by voldemortensen. View the logs.

2 weeks ago

#8 @netweb
2 weeks ago

In todays #core dev-chat a discussion arose on the feasibility of including new files in a minor release, e.g. 4.9.5, historically this has been avoided due to the how auto-updates work.

dd32 Wrote:
I’d also just like to add, I’d love to see the core E2E update tests operating before any change to how we treat updates, purely because a lot of it is based on unknowns and the past 4 years of experiences.

dd32 Wrote: re: 4.9.5 including new files:

  • I’m not going to prevent that happening, but I’ll repeat my previous stance that automatic updates wasn’t designed with that in mind, and was specifically built at a time when minor releases would only ever be bugfix releases with no new files.
  • There were technical reasons why we only did minor releases and not major ones too - adding new files is effectively a major release when you’re looking at it from an upgrade point of view.
  • The largest failing case for autoupdates at present is files_not_writable where WordPress can’t modify existing files, let alone create new files (and this is after it’s already aborted because it knew it couldn’t write to files).
  • Adding new files has a chance that a %age of users may no longer receive security auto updates afterwards when the update fails.
  • Sites may also fatal error if new files are relied upon from another file and the autoupdate couldn’t create the files - we’ve never really tested the rollback-on-fail code as it’s only been triggered a handful of times.
  • There’s also some environments where if an autoupdate adds a file, the following user-initiated updates using FTP may fail as it’ll be unable to update the file - that’s one of the reasons we specified no-new-files.
  • If I had to guess, the last two use-cases would be in the region of 0.5%~5% of users each.
  • It’s also worth reminding that in the event the autoupdate to 4.9.5 fails, it’ll stay on 4.9.4 until the owner updates the site (much like we’ve got with 4.9.3 right now) we won’t be able to tell that site to update again in any manner, so even a autoupdate would be ignored.

Further backgrount can be read (including some detatiled statsistics) here: https://make.wordpress.org/core/2017/03/11/continuing-inline-docs-improvements-adjacent-to-4-8/#comment-32248 , here's my quick extract of the key points:

Generally, update statistics have showed a clear correlation between failures and I/O operations. This is clearly seen in the difference between minor releases of different sizes, and even more pronounced when comparing those to a major release like 4.7 (with hundreds of files changed).

Additionally, we run these updates using partial build packages. Every extra file adds a significant amount of KB to the package, which adds up pretty quickly. This not only stretches the wordpress.org load balancers (when suddenly millions of sites are updating within an hour), but also each individual site, which must download the ZIP, which takes time (partial builds made things, on many shared servers, go from minutes to seconds) and introduces lots of possibilities of download failures, and thus sites needing to retry later (and wait longer) to get patched.

#9 @netweb
2 weeks ago

Great start @voldemortensen, Perl is not my go to language but I get the gist of what is going on 👍

#10 @audrasjb
7 days ago

As far as I know, it still needs a 2nd opinion / patch before landing in the next minor.
4.9.5 beta release is planned for next Tuesday so we'll need a patch in the next few days otherwise it will be punted to 4.9.6 (which is not that bad).

#11 @voldemortensen
7 days ago

  • Milestone changed from 4.9.5 to 4.9.6

Hi @audrasjb, there is nothing to be merged into core here. From the current approach, this will live outside of core, moving to 4.9.6 is fine. I think this should also be moved from a defect to a blessed task since it will likely take a little while to reach a consensus and nail down the details.

#12 @pento
7 days ago

  • Keywords 2nd-opinion removed
  • Milestone changed from 4.9.6 to WordPress.org
  • Type changed from defect (bug) to task (blessed)

Yah, this testing is going to be an ongoing project, rather than a specific core feature.

Let's leave this ticket open until we have something up and running, then it can be co-ordinated on that repo.

Note: See TracTickets for help on using tickets.