WordPress.org

Make WordPress Core

Opened 5 weeks ago

Closed 4 days ago

#40533 closed task (blessed) (fixed)

Avoid test skipping for situations that are actually errors or failures

Reported by: johnbillion Owned by:
Milestone: 4.8 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: needs-patch
Focuses: Cc:

Description

Some tests contain calls to markTestSkipped() as a result of situations such as incorrect test data or unsuitable test environments. If the test data isn't valid or the environment is not capable of supporting the test then the test should fail, otherwise the tests are not reliable.

This is a catch-all ticket for attempting to remove a bunch of test skipping that falls outside of the is_multisite() skipping that's being handled in #40531.

Change History (17)

#1 @johnbillion
5 weeks ago

In 40517:

Canonical: Don't skip tests if the test data is invalid.

See #40533

#2 @johnbillion
5 weeks ago

In 40519:

Build/Test Tools: Don't skip tests when php.net or dev.mysql.com are unreachable.

This sort of behaviour subtly hides the fact that these tests will never fail if the URLs where version number information is fetched from go away for one reason or another.

See #40533

#3 @johnbillion
5 weeks ago

In 40521:

Build/Test Tools: Remove a useless test.

See #33898, #40533

#4 @johnbillion
5 weeks ago

In 40523:

Build/Test Tools: Remove ancient UT ticket handling.

See #40533

#5 @johnbillion
5 weeks ago

In 40524:

Build/Test Tools: Remove more skipped tests that should actually be failures if their conditions aren't satisfied.

See #40533

#6 @johnbillion
5 weeks ago

In 40525:

Build/Test Tools: Replace test skipping with actual assertions when dealing with the DISALLOW_UNFILTERED_HTML, DISALLOW_FILE_MODS, and DISALLOW_FILE_EDIT constants.

If any of these constants are set during testing, they should cause the tests to fail, not to be skipped, otherwise we have a situation where failure conditions are never seen.

See #40533

#7 @johnbillion
5 weeks ago

In 40526:

Build/Test Tools: Don't trigger a skipped test when the built version of wp-embed.min.js isn't present.

The grunt command assertion prior to the check for the existence of the file is enough to safeguard this test.

See #34698, #40533

#8 @johnbillion
5 weeks ago

In 40527:

Build/Test Tools: Enable verbose mode in PHPUnit so we can see which tests are being skipped, and now that the number of skipped tests has been lowered.

See #40533, #40531

#9 @johnbillion
5 weeks ago

In 40528:

Build/Test Tools: Add some locale debugging to the Travis config so we can determine which locales are available to test with.

See #40533, #19861

#10 @johnbillion
5 weeks ago

In 40529:

Build/Test Tools: Partially revert [40524].

See #40533, #40535

#11 @johnbillion
5 weeks ago

In 40531:

Build/Test Tools: Convert more test skipping into hard failures. These dependencies should all be present when testing.

See #40533

#12 @johnbillion
5 weeks ago

In 40532:

Build/Test Tools: Convert more test skipping into hard failures. These dependencies should all be present when testing.

See #40533

#13 @johnbillion
5 weeks ago

In 40533:

Build/Test Tools: Remove unnecessary checks and skips that should instead cause failures if they ever fail.

See #40533

#14 @johnbillion
5 weeks ago

In 40544:

Build/Test Tools: Remove more unnecessary test skipping when erroneous situations occur.

See #40533

This ticket was mentioned in Slack in #core by obenland. View the logs.


7 days ago

#16 @obenland
7 days ago

@johnbillion Can this be closed for 4.8?

#17 @johnbillion
4 days ago

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.