Investigate Tests_Query::test_tag_queried_object() failures in Multisite

[SergeyBiryukov]

phpunit -c tests/phpunit/multisite.xml --group 26627 fails every other time.

​https://gist.github.com/tierra/10071429
​https://irclogs.wordpress.org/chanlog.php?channel=wordpress-dev&day=2014-04-07&sort=asc#m827715
​https://irclogs.wordpress.org/chanlog.php?channel=wordpress-dev&day=2014-04-08&sort=asc#m827777

[bpetty]

To summarize chat and the gist linked...

$ phpunit -c tests/phpunit/multisite.xml --group 26627
Installing...
Installing network...
Running as multisite...
Not running ajax tests... To execute these, use --group ajax.
PHPUnit 3.7.28 by Sebastian Bergmann.
 
Configuration read from /home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/multisite.xml
 
FF
 
Time: 3.33 seconds, Memory: 62.00Mb
 
There were 2 failures:
 
1) Tests_Query::test_tag_queried_object
Failed asserting that false is true.
 
/home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/tests/query.php:132
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/plugin.php:546
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/query.php:2186
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/query.php:3636
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/class-wp.php:541
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/class-wp.php:612
/home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/includes/testcase.php:196
/home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/tests/query.php:118
 
2) Tests_Query_TaxQuery::test_cat_uncat_action_tax
Failed asserting that false is true.
 
/home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/tests/query.php:132
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/plugin.php:546
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/query.php:2186
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/query.php:3636
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/class-wp.php:541
/home/bryan/Projects/wp-vagrant/wordpress/src/wp-includes/class-wp.php:612
/home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/includes/testcase.php:196
/home/bryan/Projects/wp-vagrant/wordpress/tests/phpunit/tests/query/taxQuery.php:120
 
FAILURES!
Tests: 2, Assertions: 2, Failures: 2.

See also #26627 and [27511] where those tests were added, and from where they started failing (i.e. they never passed in multisite even when initially written).

Some important observations from this:

• These tests pass in multisite when running with --group taxonomy.
• When isolated to --group 26627, they pass every other run, so in order to reproduce, you must run these tests more than once sometimes.

As @SergeyBiryukov has noted: When test_tag_queried_object() passes, the URL is http://example.org/?tag=tag-slug-26627, which translates into [tag] => tag-slug-26627. When it fails, the URL is http://example.org/blog/tag/tag-slug-26627/, which translates into [pagename] => blog/tag/tag-slug-26627. I've noticed that Tests_Query_TaxQuery::setUp() updates permalink_structure and never restores the original value, but for some reason restoring it in tearDown() doesn't help.

As these tests fail every other time due to permalink_structure settings, these aren't the only unit tests that fail like this in multisite. A couple data sets in Test_Canonical also exhibit this behavior (notably from #15256), and also Tests_Link::test_wp_get_shortlink does this too. To reproduce these, also run the following tests more than once:

$ phpunit -c tests/phpunit/multisite.xml --group canonical
$ phpunit -c tests/phpunit/multisite.xml --group link

Most likely, we could provide a (mostly) simple fix by ensuring that permalink_structure is configured appropriately for the tests before running them, but the more important issue we need to figure out is why this setting is persistently changing between test runs, and why it isn't being reset for each test.

[bpetty]

Also add the following unit tests to the list of related tests that fail every other run due to permalink_structure changes with Tests_User_ListAuthors between runs: test_wp_list_authors_feed, test_wp_list_authors_feed_image, and test_wp_list_authors_feed_type.

$ phpunit -c tests/phpunit/multisite.xml --group author

[slimndap]

Did you guys figure out why this happens? I ran into the same problem with an automated test for my plugin:

Test is failing on multisite every other time. Happens when using get_echo( 'the_content' ); for a single custom post type.

Error:

Trying to get property of non-object

/srv/www/wordpress-develop/src/wp-includes/post-template.php:265

[SergeyBiryukov]

Related: #28692

[jeremyfelt]

I believe this is resolved via [28967] for #28706. The permalink structure is now reset and flushed on test setup.

The full test suite is passing for me and I can run group 26627 multiple times without issue.