Make WordPress Core

Opened 7 years ago

Closed 7 years ago

Last modified 4 years ago

#40533 closed task (blessed) (fixed)

Avoid test skipping for situations that are actually errors or failures

Reported by: johnbillion's profile johnbillion Owned by:
Milestone: 4.8 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: needs-patch
Focuses: Cc:


Some tests contain calls to markTestSkipped() as a result of situations such as incorrect test data or unsuitable test environments. If the test data isn't valid or the environment is not capable of supporting the test then the test should fail, otherwise the tests are not reliable.

This is a catch-all ticket for attempting to remove a bunch of test skipping that falls outside of the is_multisite() skipping that's being handled in #40531.

Change History (20)

#1 @johnbillion
7 years ago

In 40517:

Canonical: Don't skip tests if the test data is invalid.

See #40533

#2 @johnbillion
7 years ago

In 40519:

Build/Test Tools: Don't skip tests when or are unreachable.

This sort of behaviour subtly hides the fact that these tests will never fail if the URLs where version number information is fetched from go away for one reason or another.

See #40533

#3 @johnbillion
7 years ago

In 40521:

Build/Test Tools: Remove a useless test.

See #33898, #40533

#4 @johnbillion
7 years ago

In 40523:

Build/Test Tools: Remove ancient UT ticket handling.

See #40533

#5 @johnbillion
7 years ago

In 40524:

Build/Test Tools: Remove more skipped tests that should actually be failures if their conditions aren't satisfied.

See #40533

#6 @johnbillion
7 years ago

In 40525:

Build/Test Tools: Replace test skipping with actual assertions when dealing with the DISALLOW_UNFILTERED_HTML, DISALLOW_FILE_MODS, and DISALLOW_FILE_EDIT constants.

If any of these constants are set during testing, they should cause the tests to fail, not to be skipped, otherwise we have a situation where failure conditions are never seen.

See #40533

#7 @johnbillion
7 years ago

In 40526:

Build/Test Tools: Don't trigger a skipped test when the built version of wp-embed.min.js isn't present.

The grunt command assertion prior to the check for the existence of the file is enough to safeguard this test.

See #34698, #40533

#8 @johnbillion
7 years ago

In 40527:

Build/Test Tools: Enable verbose mode in PHPUnit so we can see which tests are being skipped, and now that the number of skipped tests has been lowered.

See #40533, #40531

#9 @johnbillion
7 years ago

In 40528:

Build/Test Tools: Add some locale debugging to the Travis config so we can determine which locales are available to test with.

See #40533, #19861

#10 @johnbillion
7 years ago

In 40529:

Build/Test Tools: Partially revert [40524].

See #40533, #40535

#11 @johnbillion
7 years ago

In 40531:

Build/Test Tools: Convert more test skipping into hard failures. These dependencies should all be present when testing.

See #40533

#12 @johnbillion
7 years ago

In 40532:

Build/Test Tools: Convert more test skipping into hard failures. These dependencies should all be present when testing.

See #40533

#13 @johnbillion
7 years ago

In 40533:

Build/Test Tools: Remove unnecessary checks and skips that should instead cause failures if they ever fail.

See #40533

#14 @johnbillion
7 years ago

In 40544:

Build/Test Tools: Remove more unnecessary test skipping when erroneous situations occur.

See #40533

This ticket was mentioned in Slack in #core by obenland. View the logs.

7 years ago

#16 @obenland
7 years ago

@johnbillion Can this be closed for 4.8?

#17 @johnbillion
7 years ago

  • Resolution set to fixed
  • Status changed from new to closed

#18 @jnylen0
7 years ago

Each of these failures should have some information about how to resolve it. For example, the importer plugin is missing by default if you use git to develop against WordPress (#41345). This isn't documented anywhere that I could find.

Ideally the error message should describe what is going on and point to documentation that explains how to resolve the issue. For the importer test failures, we can link to

#19 @johnbillion
7 years ago

In 41305:

Build/Test Tools: Remove ancient UT ticket handling for the 4.7 branch.

See #40533

Merges [40523] to the 4.7 branch.

#20 @SergeyBiryukov
4 years ago

In 46996:

Tests: In test_readme(), if the HTTP request to or failed, include response code in the error message.

See #40533, #44613.

Note: See TracTickets for help on using tickets.