WordPress.org

Make WordPress Core

Opened 6 years ago

Last modified 15 months ago

#34694 closed enhancement

Facilitate automated testing in context of pull requests and diffs — at Version 1

Reported by: westonruter Owned by:
Milestone: Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords:
Focuses: Cc:

Description (last modified by westonruter)

Using GitHub for core development has been a wishlist item for awhile now. Individual feature plugins have been increasingly developed on GitHub (e.g. Widget Customizer, WP-API). Going an extra step, the Twenty Sixteen theme only exists on GitHub and isn't even committed to trunk. So while feature plugins and themes are often being developed primarily on GitHub, other work on Core is generally not (although this may change).

While there is no official wordpress-develop Git repo on GitHub (yet), anyone can clone the wordpress-develop Git repo and push it to GitHub. Teams working on various components can then work in the context of those GitHub clones, and then when work is complete a committer can apply the patch to SVN. Travis CI is already configured in the wordpress-develop repo, and so anyone who clones the repo to GitHub can automatically turn on Travis CI to start getting automated build checks for each commit.

Work on GitHub is usually done in the context of pull requests, and Travis CI can be especially useful when running in response to a pull request being opened or a commit pushed to a feature branch with an existing pull request. As reported in #30017, many automated tests are unnecessarily slow. This is in large part because with each commit, _all_ checks are being made across all files in the project, irrespective of whether they were recently modified or not.

When Travis CI does a build in the context of a pull request, it makes available the branch checked out and the branch being merged into, and with these two refs in hand, we can gather a list of the files (and patches) that were specifically changed. When we have these, we can optimize Travis CI to only run the checks that are relevant to the changes in the feature branch. For instance, if no PHP files were changed, then all of the jobs that run PHPUnit tests can be cancelled. If no JS files were modified, then the JSHint checks can be skipped.

It can be painful to introduce new automated checks to WordPress because they can add a lot of noise for automated tests and fixing requires a touching a lot of files, possibly invalidating many pending patches. When Travis runs in the context of a pull request (or when automated tests are run when trunk/master is not checked out), then again, there is a diff available for the specific changes. We can introduce PHP_CodeSniffer for core (#30153) when Travis runs in in the context of a pull request, and it can skip reporting errors on any lines that aren't modified in the commit. This is likewise how JSCS linting can be added (#31823).

For existing code which causes Travis CI to only report coding standard violations to lines changed, see https://github.com/xwp/wp-dev-lib/blob/master/travis.script.sh when $LIMIT_TRAVIS_PR_CHECK_SCOPE is patches.

For commits pushed to master on Travis CI (outside the context of a pull request), it may make sense to configure Travis to check for changes made since the last major release.

Change History (1)

#1 @westonruter
6 years ago

  • Description modified (diff)
Note: See TracTickets for help on using tickets.