Add browser-based testing and automated acceptance tests (BrowserStack)

[westonruter]

Automated testing in WordPress is primarily focused on PHPUnit testing, with some JavaScript QUnit tests. While the PHP tests are run against every version of PHP since 5.2, the JavaScript tests are limited to one single browser environment (PhantomJS). This has meant that browser-specific bugs have been introduced (e.g. #30781) because manual testing isn't always performed in all the browsers that WordPress supports. We should connect WordPress unit tests with something like BrowserStack to run the QUnit tests in the various browsers.

While we're at it, the unit/integration testing in PHPUnit and QUnit only take us so far in WordPress as it is. What WordPress is missing are acceptance tests that fire up an actual browser to do actual tasks that users would perform when using WordPress. Such acceptance tests would make WordPress much more resilient against regressions, and it would open up a lot of new areas for automated testing that are currently very difficult to test using unit/integration tests. For instance, the Customizer heavily uses data from both PHP and JavaScript along with Ajax requests between the two. To create unit/integration testing mocks for PHP, JS, and Ajax requests adds a lot of testing overhead and is also difficult to maintain.

I believe acceptance testing would allow much more of WordPress to be tested automatically, and it would allow tests to be written by more people.

See also ​https://wordpress.slack.com/archives/core/p1419407324003594

[westonruter]

Let's plan on this for WCUS contributor day.

[jonathanbardo]

On way we can easily add acceptance testing using BrowserStack is to use a wp-cli helper plugin by 10up ​https://github.com/10up/wp-codeception.

For VVV testing we need to install Java & the actual plugin (which enables wp-cli commands):

​https://github.com/10up/wp-codeception#install-required-node-modules-and-composer-dependencies

To be able to test this locally, we would need an account and an api key from BrowserStack (replace value in /tests/codeception/acceptance.suite.yml [we need a better way to override those]) and we also need the local tunel binaries found here: ​https://www.browserstack.com/local-testing#command-line

Here is a video of the output of the following command:

wp codeception run --debug --env=chrome

​https://cldup.com/j6NWUyppF5.mp4

We would have to do some modifications on how travis downloads, install and run the tests. We might also want to limit how much of those tests are run everytime since it takes a fair amount of time. I think @jorbin is the right person to help with this.

[rahul286]

We use NightWatch (​http://nightwatchjs.org/) for acceptance testing.

Here are test cases for one of our plugin - ​https://github.com/rtCamp/rtMedia/tree/master/tests/functional

For any project, before we automate it, we write all test cases in a tool like MozTrap(​https://github.com/mozilla/moztrap/) . Sometimes we simply use excelsheet. And then we write code for each test cases.

Below are some references for using NightWatch with BrowserStack:

​https://github.com/browserstack/automate-node-samples/tree/master/nightwatch
​http://pauldambra.github.io/2014/03/testing-with-browserstack-and-selenium.html

[jonathanbardo]

From my point of view, one of the most compelling reason to write acceptance tests using Codeception (WP-Codeception package) is how well integrated it already is with WordPress and WP-CLI. When writing tests, all WordPress functions are available for you to use and could potentially save time in the long run.

[rahul286]

I came across Codeception, but missed WP-Codeception. Thanks for sharing. :-)

When choosing a framework for acceptance test we did not look for WordPress integration as by nature, I think acceptance tests are not supposed to deal with system internals. Kind of black box testing.

So can you please help me understand if you use Codeception for acceptance testing only or for unit testing as well?

Whichever project we use to automate acceptance testing, I believe we will need another tool for test case management. Any suggestions on this?

[westonruter]

Replying to rahul286:

When choosing a framework for acceptance test we did not look for WordPress integration as by nature, I think acceptance tests are not supposed to deal with system internals. Kind of black box testing.

That makes sense to me. I don't have enough familiarity to know the advantages/disadvantages. I suppose my main concern is that the most established/standard tool be chosen for the job, one with the widest adoption. I honestly don't know how common it is for acceptance testing to be used in WP projects, so I don't know what is most commonly used.

So can you please help me understand if you use Codeception for acceptance testing only or for unit testing as well?

If using Codeception for core, I'd think we'd specifically only want to use it for acceptance testing since unit/integration tests are pretty firmly established in direct PHPUnit.

Whichever project we use to automate acceptance testing, I believe we will need another tool for test case management. Any suggestions on this?

Beyond just committing the acceptance testing scripts to the repo? Do you mean a tool for facilitating the development of the tests, like recording macros?

[jdgrimes]

FWIW: I'm just starting to explore acceptance testing for my plugins, and I've been building off of Codeception via ​WP Browser. I'm planning on using Codeception only for the acceptance tests, and sticking with pure PHPUnit for the unit tests.

[moreauf]

Replying to rahul286:

When choosing a framework for acceptance test we did not look for WordPress integration as by nature, I think acceptance tests are not supposed to deal with system internals. Kind of black box testing.

Access to internals in acceptance tests is quite important to set up fixtures. Say I want to test how some plugin's search box reacts. First I want to insert thousands of items and I definitely don't want my acceptance test to insert them "by hand".

[rahul286]

Replying to moreauf:

Access to internals in acceptance tests is quite important to set up fixtures. Say I want to test how some plugin's search box reacts. First I want to insert thousands of items and I definitely don't want my acceptance test to insert them "by hand".

Totally agree with you. We moved to CodeCeption a few months back and currently using it on our projects for acceptance tests.

Replying to westonruter:

Replying to rahul286:

Whichever project we use to automate acceptance testing, I believe we will need another tool for test case management. Any suggestions on this?

Beyond just committing the acceptance testing scripts to the repo? Do you mean a tool for facilitating the development of the tests, like recording macros?

Sorry for delayed reply. I was referring to something like TestRail ​http://www.gurock.com/testrail/ or Moztrap ​https://moztrap.readthedocs.io/. A place to manage test cases themselves in text format. People then try to automate test cases written inside test-case management tool.

This will separate the process of creating test cases and automating them.

Also, a tool such as TestRail can record every test-run. So may be we can run acceptance test cases with each SVN commit and record results in a central place. This part may be skipped by sending logs to trac or somewhere.

But I believe we will still need a test case management tool.

[westonruter]

In part given WP-CLI's embrace of Behat and also the recent introduction of @pgibbs's ​WordHat and the promise of scenarios being understandable/writable by non-developers, I wanted to try learning some Behat myself and apply it to the context of doing automated acceptance tests in the customizer. Here's the initial proof of concept I came up with: ​https://github.com/xwp/wp-js-widgets/pull/43

I can't speak Behat's to its merits over Codeception, and I don't mean to negate @jonathanbardo's 34693_first_draft.diff​.

I welcome comments and critique. Mainly I want to get this kicked off again, and specifically focus it on the specific use case of testing the features and flows being introduced in the media widget (#32417).

[jdgrimes]

Just FYI, ​Codeception also supports BDD using Gherkin.

I have no experience with Behat, so I can't comment on the pros and cons over Codeception either. I can only say that Codeception has done the job for me, and done it well. It would be good to hear from anybody that has worked with several different tools, what they have found the pros and cons to be.

[Bsport]

So I've been using Behat\WordHat lately on a project within customizer and its proved pretty plainless, Now I choose Behat\WordHat route because of the Gherkin, non-developers can write tests and understands whats meant to happen. My second deciding factor was, I wanted a more widly understood BDD testing framework and using both google trends and slideshare search result counts for behat and codeception, behat won both these, slide share by a large margin. These 2 points were then wrapped with the fact it took me all of about 2 minutes to write an acceptence test for the quick draft widget using WordHat + its out of the box wpcli support.

[lukefiretoss]

There is also these WP extensions for Codeception.
​https://github.com/lucatume/wp-browser