WordPress.org

Make WordPress Core

Opened 3 months ago

Closed 2 months ago

Last modified 4 days ago

#40533 closed task (blessed) (fixed)

Avoid test skipping for situations that are actually errors or failures

Reported by: johnbillion Owned by:
Milestone: 4.8 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: needs-patch
Focuses: Cc:

Description

Some tests contain calls to markTestSkipped() as a result of situations such as incorrect test data or unsuitable test environments. If the test data isn't valid or the environment is not capable of supporting the test then the test should fail, otherwise the tests are not reliable.

This is a catch-all ticket for attempting to remove a bunch of test skipping that falls outside of the is_multisite() skipping that's being handled in #40531.

Change History (18)

#1 @johnbillion
3 months ago

In 40517:

Canonical: Don't skip tests if the test data is invalid.

See #40533

#2 @johnbillion
3 months ago

In 40519:

Build/Test Tools: Don't skip tests when php.net or dev.mysql.com are unreachable.

This sort of behaviour subtly hides the fact that these tests will never fail if the URLs where version number information is fetched from go away for one reason or another.

See #40533

#3 @johnbillion
3 months ago

In 40521:

Build/Test Tools: Remove a useless test.

See #33898, #40533

#4 @johnbillion
3 months ago

In 40523:

Build/Test Tools: Remove ancient UT ticket handling.

See #40533

#5 @johnbillion
3 months ago

In 40524:

Build/Test Tools: Remove more skipped tests that should actually be failures if their conditions aren't satisfied.

See #40533

#6 @johnbillion
3 months ago

In 40525:

Build/Test Tools: Replace test skipping with actual assertions when dealing with the DISALLOW_UNFILTERED_HTML, DISALLOW_FILE_MODS, and DISALLOW_FILE_EDIT constants.

If any of these constants are set during testing, they should cause the tests to fail, not to be skipped, otherwise we have a situation where failure conditions are never seen.

See #40533

#7 @johnbillion
3 months ago

In 40526:

Build/Test Tools: Don't trigger a skipped test when the built version of wp-embed.min.js isn't present.

The grunt command assertion prior to the check for the existence of the file is enough to safeguard this test.

See #34698, #40533

#8 @johnbillion
3 months ago

In 40527:

Build/Test Tools: Enable verbose mode in PHPUnit so we can see which tests are being skipped, and now that the number of skipped tests has been lowered.

See #40533, #40531

#9 @johnbillion
3 months ago

In 40528:

Build/Test Tools: Add some locale debugging to the Travis config so we can determine which locales are available to test with.

See #40533, #19861

#10 @johnbillion
3 months ago

In 40529:

Build/Test Tools: Partially revert [40524].

See #40533, #40535

#11 @johnbillion
3 months ago

In 40531:

Build/Test Tools: Convert more test skipping into hard failures. These dependencies should all be present when testing.

See #40533

#12 @johnbillion
3 months ago

In 40532:

Build/Test Tools: Convert more test skipping into hard failures. These dependencies should all be present when testing.

See #40533

#13 @johnbillion
3 months ago

In 40533:

Build/Test Tools: Remove unnecessary checks and skips that should instead cause failures if they ever fail.

See #40533

#14 @johnbillion
3 months ago

In 40544:

Build/Test Tools: Remove more unnecessary test skipping when erroneous situations occur.

See #40533

This ticket was mentioned in Slack in #core by obenland. View the logs.


2 months ago

#16 @obenland
2 months ago

@johnbillion Can this be closed for 4.8?

#17 @johnbillion
2 months ago

  • Resolution set to fixed
  • Status changed from new to closed

#18 @jnylen0
4 days ago

Each of these failures should have some information about how to resolve it. For example, the importer plugin is missing by default if you use git to develop against WordPress (#41345). This isn't documented anywhere that I could find.

Ideally the error message should describe what is going on and point to documentation that explains how to resolve the issue. For the importer test failures, we can link to https://make.wordpress.org/core/handbook/contribute/git/#unit-tests.

Note: See TracTickets for help on using tickets.