phpunit fails w/ a fatal error when using the multisite config

[Denis-de-Bernardy]

On a clean install, fails in a couple of places, and eventually runs into a fatal error when it tries to call ->exists() on a WP_Error instead of a WP_User.

On a non-clean install, it succeeds but eventually stops outright, complaining that a site is installed already.

[nacin]

Our PHP tests work just fine for me under multisite. We could benefit from more information. What are these errors? What's the backtrace? Why are we randomly patching core to fix them?

I don't know what "clean" or "non-clean" means. Our tests are always designed to run on their own database.

[Denis-de-Bernardy]

log output after applying the diff I just posted.

[nacin]

You are making me want to put a minimum word count on your tickets, Denis.

[Denis-de-Bernardy]

Replying to nacin:

Our PHP tests work just fine for me under multisite. We could benefit from more information. What are these errors? What's the backtrace? Why are we randomly patching core to fix them?

Try the git version you posted no the dev blog. And we're not randomly patching core, I merely posted what seemed to allow to run the tests to their completion — with lots of fails and errors. (See the log I attached.)

I don't know what "clean" or "non-clean" means. Our tests are always designed to run on their own database.

But then, when the tests don't complete and run into a fatal error, some cleaning isn't properly done, hence the "unclean" wording vs a "clean" drop/create database.

[Denis-de-Bernardy]

log (and trace) without the patch applied

[Denis-de-Bernardy]

Replying to nacin:

You are making me want to put a minimum word count on your tickets, Denis.

Whatever… I've posted two logs: with and without the diff I attached.

[nacin]

It appears the cause of this error is the user factory not returning a user. I don't know why. We're not going to use the patch you provided as it does not make sense to patch the symptom in core if there's a bug or race condition in the test suite. It would be better if we knew the contents of the WP_Error. Usually those actually contain things like information about the error.

[jdgrimes]

See #24800, especially my comment here. I have a patch there, but I guess it got overlooked. This was caused by [27041].

[SergeyBiryukov]

Confirmed that [27086] fixed the fatal error here.

[Denis-de-Bernardy]

There's still a problem when you run the tests without dropping/re-creating the database:

phpunit -c tests/phpunit/multisite.xml
Installing...
Installing network...
Running as multisite...
Not running ajax tests... To execute these, use --group ajax.
PHPUnit 3.7.31 by Sebastian Bergmann.

Configuration read from /Users/denis/wk/wpdevel/tests/phpunit/multisite.xml

.............................................................   61 / 1966 (  3%)
............SS...SSSSSS.............S...SS............SS.....  122 / 1966 (  6%)
...........S............S.................................S..  183 / 1966 (  9%)
...................SSS.......SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS  244 / 1966 ( 12%)
S..S........................S............................S...  305 / 1966 ( 15%)
................S...........................S.SSSSSSSSS......  366 / 1966 ( 18%)
............................S..................S.......I.....  427 / 1966 ( 21%)
....................................................SSSS.S..S  488 / 1966 ( 24%)
SSSS.........................................................  549 / 1966 ( 27%)
.............................................................  610 / 1966 ( 31%)
.............................................................  671 / 1966 ( 34%)
.............................................................  732 / 1966 ( 37%)
.............................................................  793 / 1966 ( 40%)
..........................S..................................  854 / 1966 ( 43%)
.............................................................  915 / 1966 ( 46%)
.............................................................  976 / 1966 ( 49%)
..............................SSSSSS.........S.....S......... 1037 / 1966 ( 52%)
......................................<h1>Already Installed</h1><p>You appear to have already installed WordPress. To reinstall please clear your old database tables first.</p></body></html>

After clearing the DB, the tests still fail in two locations:

1) Tests_Canonical::test with data set #0 ('?cat=%d', '/category/parent/', 15256)
Ticket #15256
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'/category/parent/'
+'/blog/category/parent/'

/Users/denis/wk/wpdevel/tests/phpunit/tests/canonical.php:133

2) Tests_Link::test_wp_get_shortlink
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'http://wptest.org/blog/2014/02/04/post-title-1/'
+'http://wptest.org/?p=1213'

/Users/denis/wk/wpdevel/tests/phpunit/tests/link.php:36

FAILURES!
Tests: 1920, Assertions: 9503, Failures: 2, Incomplete: 1, Skipped: 116.

[jdgrimes]

Replying to Denis-de-Bernardy:

There's still a problem when you run the tests without dropping/re-creating the database:

Yes, I had experienced that while debugging this as well. Although wp-tests-config-sample.php claims that all tables with the prefix will be dropped, that isn't true. (It may have been before, I don't know.) Only the tables for the main site on the network are dropped, and then only those which core creates. So if for some reason the changes get committed to MySQL, you may have to clean a second site's tables from the database manually. Normally this will always ROLLBACK though.

[bpetty]

Those two test failures have existed (and been known about) for well over 6 months. Note that they sometimes also don't fail. Would certainly love fixes for them if you can find it, but please do so in a new ticket, not on this one.

Clearing the entire DB is kind of low priority at the moment as well since they are cleaned up assuming the unit tests don't generate fatal errors like they were here. Just clear and move on for now. That's also an issue for another ticket.