WordPress.org

Make WordPress Core

Opened 12 months ago

Closed 10 months ago

Last modified 10 months ago

#48145 closed defect (bug) (fixed)

Random PHP test failures

Reported by: SergeyBiryukov Owned by: SergeyBiryukov
Milestone: 5.3.2 Priority: low
Severity: normal Version:
Component: Build/Test Tools Keywords: fixed-major
Focuses: Cc:

Description (last modified by SergeyBiryukov)

Every few commits, there's now a random PHP 7.1 test failure on Travis, mostly off-by-one errors unrelated to the preceding commit. Started about 1 month ago, after August 20th.

Seems like either PHP 7.1 tests are for some reason more prone to random errors than other versions, or it's just more prominent now that random date/time failures from #45821 are fixed.

Creating a ticket just for reference, any ideas as to why this happens appreciated.

Some examples:

1) WP_Test_REST_Posts_Controller::test_get_items_search_query
Failed asserting that 6 matches expected 7.
/var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:459
1) WP_Test_REST_Posts_Controller::test_get_items_pagination_headers
Failed asserting that 49 matches expected 50.
/var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:1157
1) Tests_Canonical_PageOnFront::test with data set #3 ('/?page=1', '/')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'/'
+'/?page=1'
/var/www/tests/phpunit/includes/testcase-canonical.php:268
/var/www/tests/phpunit/tests/canonical/pageOnFront.php:39

2) Tests_Query_MetaQuery::test_meta_query_single_query_compare_arithmetic_comparisons
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 2319
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:658
/var/www/tests/phpunit/tests/query/metaQuery.php:162
1) Tests_Query::test_tag_querystring_single_term
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => 2022
-    1 => 2023
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:658
/var/www/tests/phpunit/tests/query.php:321
1) Tests_Term_getTerms::test_wp_delete_term_should_invalidate_cache
Failed asserting that 1 matches expected 2.
/var/www/tests/phpunit/tests/term/getTerms.php:181
1) Tests_Query_VerbosePageRules::test_page_page_2
is_404 is true but is expected to be false. 
is_page is false but is expected to be true. 
is_paged is false but is expected to be true. 
is_singular is false but is expected to be true.
/var/www/tests/phpunit/includes/abstract-testcase.php:971
/var/www/tests/phpunit/tests/query/conditionals.php:327

1) Tests_Query_MetaQuery::test_meta_query_decimal_results
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 2414
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:658
/var/www/tests/phpunit/tests/query/metaQuery.php:1170
1) Tests_Term_getTerms::test_hierarchical_true_with_parent
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 1277
-    1 => 1280
+    0 => 1280
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:658
/var/www/tests/phpunit/tests/term/getTerms.php:2257
1) Tests_Query_SetupPostdata::test_secondary_query_nextpage
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 (
-    0 => 'Post 1 Page 1'
-    1 => 'Post 1 Page 2'
+    0 => 'Post 2 Page 1'
+    1 => 'Post 2 Page 2'
 )
/var/www/tests/phpunit/tests/query/setupPostdata.php:249
1) Tests_Query_MetaQuery::test_meta_between_not_between
Failed asserting that 1 matches expected 2.
/var/www/tests/phpunit/tests/query/metaQuery.php:1537
1) Tests_Post_getPages::test_get_pages_non_hierarchical_non_empty_child_of
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => 1677
     1 => 1678
-    2 => 1679
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:649
/var/www/tests/phpunit/tests/post/getPages.php:617
1) Tests_Post_Query::test_orderby_post__in_array
Failed asserting that two arrays are identical.
--- Expected
+++ Actual
@@ @@
 Array &0 (
-    0 => 428
-    1 => 426
-    2 => 429
+    0 => 426
+    1 => 429
 )
/var/www/tests/phpunit/tests/post/query.php:192
1) Tests_Get_Archives::test_wp_get_archives_limit
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'<li><a href='http://example.org/?p=1143'>Post title 3347</a></li>\n
-	<li><a href='http://example.org/?p=1142'>Post title 3346</a></li>\n
+'<li><a href='http://example.org/?p=1142'>Post title 3346</a></li>\n
 	<li><a href='http://example.org/?p=1141'>Post title 3345</a></li>\n
 	<li><a href='http://example.org/?p=1140'>Post title 3344</a></li>\n
-	<li><a href='http://example.org/?p=1139'>Post title 3343</a></li>'
+	<li><a href='http://example.org/?p=1139'>Post title 3343</a></li>\n
+	<li><a href='http://example.org/?p=1138'>Post title 3342</a></li>'
/var/www/tests/phpunit/tests/functions/getArchives.php:79

2) Tests_Get_Archives::test_wp_get_archives_show_post_count
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'<li><a href='http://example.org/?m=201909'>September 2019</a>&nbsp;(8)</li>'
+'<li><a href='http://example.org/?m=201909'>September 2019</a>&nbsp;(7)</li>'
/var/www/tests/phpunit/tests/functions/getArchives.php:127
1) Tests_Query_MetaQuery::test_meta_query_compare_not_exists
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => 990
     1 => 991
-    2 => 992
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:649
/var/www/tests/phpunit/tests/query/metaQuery.php:654
1) Test_WP_Customize_Nav_Menus::test_load_available_items_query_returns_post_item_with_page_number
Failed asserting that an array contains Array &0 (
    'id' => 'post-1030'
    'title' => 'Post Title'
    'type' => 'post_type'
    'type_label' => 'Post'
    'object' => 'post'
    'object_id' => 1030
    'url' => 'http://example.org/?p=1030'
).
/var/www/tests/phpunit/tests/customize/nav-menus.php:182
1) Tests_Comment_Template::test_get_comments_number_text_declension_with_custom_args with data set #10 (2, '% Comments', '2 Comments')
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'2 Comments'
+'No Comments'
/var/www/tests/phpunit/tests/comment/template.php:68
1) Tests_oEmbed_Discovery::test_add_oembed_discovery_links_to_attachment
is_404 is true but is expected to be false. 
is_attachment is false but is expected to be true. 
is_single is false but is expected to be true. 
is_singular is false but is expected to be true.
/var/www/tests/phpunit/includes/abstract-testcase.php:964
/var/www/tests/phpunit/tests/oembed/discovery.php:81
1) Tests_Rewrite::test_url_to_postid_hierarchical_with_matching_leaves
Failed asserting that 0 matches expected 3429.
/var/www/tests/phpunit/tests/rewrite.php:245
1) Tests_Post_getPages::test_get_pages_meta
Failed asserting that 0 matches expected 1.
/var/www/tests/phpunit/tests/post/getPages.php:267
1) Tests_Link_GetNextCommentsLink::test_page_should_respect_value_of_cpage_query_var
PHPUnit\Framework\Exception: Argument #2 (No Value) of PHPUnit\Framework\Assert::assertContains() must be a array, traversable or string
/var/www/tests/phpunit/tests/link/getNextCommentsLink.php:19
1) Tests_Term_WpSetObjectTerms::test_set_object_terms_by_id
Failed asserting that 4 matches expected 5.
/var/www/tests/phpunit/tests/term/wpSetObjectTerms.php:128
2) Tests_Term_WpSetObjectTerms::test_set_object_terms_by_name
Failed asserting that 4 matches expected 5.
/var/www/tests/phpunit/tests/term/wpSetObjectTerms.php:161
1) Tests_Comment_Submission::test_submitting_comment_with_content_too_long_returns_error
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'comment_content_column_length'
+'comment_on_draft'
/var/www/tests/phpunit/tests/comment-submission.php:687
1) Test_WP_Customize_Nav_Menu_Item_Setting::test_save_updated
Failed asserting that an array is not empty.
/var/www/tests/phpunit/tests/customize/nav-menu-item-setting.php:658
1) Tests_Query_Conditionals::test_is_single_with_attachment
Failed asserting that false is true.
/var/www/tests/phpunit/tests/query/conditionals.php:1025
1) Tests_Query_MetaQuery::test_meta_query_compare_exists_with_value_should_convert_to_equals
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
-    0 => 985
 )
/var/www/tests/phpunit/includes/abstract-testcase.php:684
/var/www/tests/phpunit/tests/query/metaQuery.php:593
1) WP_Test_REST_Posts_Controller::test_get_items_author_query
Failed asserting that 31 matches expected 32.
/var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:269
2) WP_Test_REST_Posts_Controller::test_get_items_author_exclude_query
Failed asserting that 31 matches expected 32.
/var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:301
3) WP_Test_REST_Posts_Controller::test_get_items_search_query
Failed asserting that 30 matches expected 31.
/var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:510
...
1) Tests_Post::test_wp_count_posts_filtered
Failed asserting that '2' matches expected 3.
/var/www/tests/phpunit/tests/post.php:836
1) Tests_Rewrite_OldSlugRedirect::test_old_slug_doesnt_redirect_when_reused
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'http://example.org/?p=3483/'
+'http://example.org/foo-bar/'
/var/www/tests/phpunit/tests/rewrite/oldSlugRedirect.php:144
1) Tests_Rewrite_AddRewriteEndpoint::test_permalink_endpoint_only_applies_on_permalink
Failed asserting that false is true.
/var/www/tests/phpunit/tests/rewrite/addRewriteEndpoint.php:88
1) Tests_Post::test_wp_count_posts
Failed asserting that 0 matches expected 1.
/var/www/tests/phpunit/tests/post.php:820
1) Tests_Query_Conditionals::test_main_comments_feed
is_404 is true but is expected to be false. 
is_comment_feed is false but is expected to be true. 
is_feed is false but is expected to be true. 
is_single is false but is expected to be true. 
is_singular is false but is expected to be true.
/var/www/tests/phpunit/includes/abstract-testcase.php:964
/var/www/tests/phpunit/tests/query/conditionals.php:430

Change History (35)

#1 @johnbillion
12 months ago

Might be due to the sort order being indeterminate. If posts are sorted by a field other than ID then the field value may not be unique (eg. two identical dates), which makes the sort order indeterminate.

Not sure why this would start showing up now though.

#2 @SergeyBiryukov
12 months ago

  • Description modified (diff)

#3 @SergeyBiryukov
12 months ago

  • Description modified (diff)

#4 @SergeyBiryukov
12 months ago

  • Description modified (diff)

#5 @desrosj
11 months ago

  • Keywords needs-patch added
  • Milestone changed from Awaiting Review to 5.4
  • Priority changed from normal to low

Let's investigate and fix these after 5.3 is branched.

#6 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#7 @SergeyBiryukov
11 months ago

It looks like most of the failures here have to do with $this->factory->post->create() failing to create a post and the tests not handling that appropriately. Perhaps adding some logging there would help?

#8 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#9 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#10 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#11 @SergeyBiryukov
11 months ago

In 46657:

REST API: Speed up pagination unit tests by creating less fixtures and reusing them where possible.

Includes minor documentation and code layout fixes for better readability.

See #30017, #48145.

#12 @desrosj
11 months ago

  • Summary changed from Random PHP 7.1 test failures to Random PHP test failures

Looks like these have creeped into 7.4 builds:

1) WP_Test_REST_Posts_Controller::test_get_items_include_query

Failed asserting that 1691 matches expected 1692.

/var/www/tests/phpunit/tests/rest-api/rest-posts-controller.php:343

This ticket was mentioned in Slack in #core by desrosj. View the logs.


11 months ago

#14 @johnbillion
11 months ago

In 46670:

REST API: Use explicit dates in a post ordering test to avoid indeterminate ordering.

See #48145

#15 @SergeyBiryukov
11 months ago

In 46671:

Coding Standards: Fix WPCS issues in [46670].

See #48145.

#16 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#17 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#18 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#19 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#20 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#21 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#22 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#23 @SergeyBiryukov
11 months ago

  • Description modified (diff)

#24 @SergeyBiryukov
11 months ago

  • Description modified (diff)

This ticket was mentioned in Slack in #core by sergey. View the logs.


11 months ago

#26 @SergeyBiryukov
11 months ago

  • Description modified (diff)

This ticket was mentioned in Slack in #core by sergey. View the logs.


11 months ago

#28 @SergeyBiryukov
10 months ago

In 46829:

Tests: Speed up comment submission unit tests by creating less fixtures and reusing them where possible.

See #30017, #48145.

#29 @SergeyBiryukov
10 months ago

In 46830:

Tests: Pad incrementors used in factories with leading zeros to make sure MySQL sorting works as expected.

Previously, tests could create sequential objects with names like Term 99, Term 100, which by default would be returned in reverse order when sorted by name alphabetically, causing seemingly random test failures that are extremely hard to debug.

See #48145.

#30 @SergeyBiryukov
10 months ago

In 46831:

Tests: Make sure network factory returns an integer ID for created objects, for consistency with other factories and the documented value.

See #48145.

#31 @SergeyBiryukov
10 months ago

  • Owner set to SergeyBiryukov
  • Resolution set to fixed
  • Status changed from new to closed

In 46968:

Date/Time: In wp_insert_post(), when checking the post date to set future or publish status, use a proper delta comparison.

[3525] allowed a difference up to 59 seconds between the post date/time and the current time to consider the post published instead of scheduled, but that didn't take start of a new minute into account.

Rapidly creating post fixtures in unit tests could encounter a one-second discrepancy between current_time( 'mysql' ) and gmdate( 'Y-m-d H:i:s' ), returning values like 2019-12-16 23:43:00 vs. 2019-12-16 23:42:59, respectively, and setting the post to a future status instead of publish.

[45851], while working as intended, made the issue somewhat more likely to occur.

This caused all sorts of occasional random failures in various tests on Travis, mostly on PHP 7.1.

Fixes #48145.

#32 @SergeyBiryukov
10 months ago

  • Keywords fixed-major added; needs-patch removed
  • Milestone changed from 5.4 to 5.3.2
  • Resolution fixed deleted
  • Status changed from closed to reopened

[46968] should be backported to the 5.3 branch.

#33 @SergeyBiryukov
10 months ago

In 46969:

Date/Time: Add a unit test for [46968].

See #48145.

#34 @SergeyBiryukov
10 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 46975:

Date/Time: In wp_insert_post(), when checking the post date to set future or publish status, use a proper delta comparison.

[3525] allowed a difference up to 59 seconds between the post date/time and the current time to consider the post published instead of scheduled, but that didn't take start of a new minute into account.

Rapidly creating post fixtures in unit tests could encounter a one-second discrepancy between current_time( 'mysql' ) and gmdate( 'Y-m-d H:i:s' ), returning values like 2019-12-16 23:43:00 vs. 2019-12-16 23:42:59, respectively, and setting the post to a future status instead of publish.

[45851], while working as intended, made the issue somewhat more likely to occur.

This caused all sorts of occasional random failures in various tests on Travis, mostly on PHP 7.1.

Merges [46968] and [46969] to the 5.3 branch.
Fixes #48145.

This ticket was mentioned in Slack in #core-datetime by sergey. View the logs.


10 months ago

Note: See TracTickets for help on using tickets.