Tests_Embed_Template tests fail when run in isolation

[jrf]

When running the Tests_Embed_Template test in isolation, two of the tests fail.

phpunit --filter Tests_Embed_Template

#) Tests_Embed_Template::test_oembed_output_post_with_thumbnail
DOMDocument::loadHTML(): AttValue: " expected in Entity, line: 10

path/to/wp/tests/phpunit/tests/oembed/template.php:66

#) Tests_Embed_Template::test_oembed_output_attachment
DOMDocument::loadHTML(): AttValue: " expected in Entity, line: 10

path/to/wp/tests/phpunit/tests/oembed/template.php:110

Accidentally discovered when I added the import group to the <groups> <excludes> and ran the complete test suite and those two tests - which are unrelated to the import tests - started erroring out.

A tentative conclusion from that can be that:

1. The Tests_Embed_Template tests have an undeclared dependency on the tests in the import group.
2. The Tests_Embed_Template tests only pass by accident due to the order in which the tests are run, which is not guaranteed and can be influenced in more recent PHPUnit versions, so cannot be relied upon.

Also see: ​https://phpunit.readthedocs.io/en/stable/configuration.html#the-executionorder-attribute

This needs further investigation.

Solution direction: The test should be made independent of the import tests.

Originally reported in https://core.trac.wordpress.org/ticket/53363#comment:39

[SergeyBiryukov]

Thanks for the ticket!

Just noting that I cannot seem to reproduce yet, these tests run as expected for me:

S:\home\wordpress.test\develop-trunk>phpunit --filter Tests_Embed_Template
Installing...
Running as single site... To run multisite, use -c tests/phpunit/multisite.xml
Not running ajax tests. To execute these, use --group ajax.
Not running ms-files tests. To execute these, use --group ms-files.
Not running external-http tests. To execute these, use --group external-http.
PHPUnit 7.5.20 by Sebastian Bergmann and contributors.

.................                                                 17 / 17 (100%)

You should really fix these slow tests (>150ms)...
 1. 303ms to run Tests_Embed_Template:test_oembed_output_post


Time: 53.52 seconds, Memory: 144.00 MB

OK (17 tests, 55 assertions)

That said, there might indeed be something unstable about these tests that needs further investigation.

[costdev]

Tests to reproduce

Env

PHP: 7.4.18
PHPUnit: 9.5.10
Localhost: WSL2 with Ubuntu 18.04
OS: Windows 10

In Isolation (--filter Tests_Embed_Template)

1. import added to groups excludes.
2. 'import' => true, added to $skipped_groups in bootstrap.php.
3. Both of the above added.

Complete Test Suite

1. import added to groups excludes.
2. 'import' => true, added to $skipped_groups in bootstrap.php.
3. Both of the above added.

Results

Couldn't reproduce - all tests pass.

---

What PHPUnit version are you running?

[jrf]

@costdev Thanks for trying to reproduce. I can currently reproduce it even without the exclude for the import group and the standard config.

Command phpunit -c phpunit.xml.dist --filter Tests_Embed_Template

Tested against PHP: 7.2.34, 7.3.31, 7.4.24, 8.0.11, 8.1.0rc2 (other PHP versions probably as well, but stopped testing)
Tested with PHPUnit: 7.5.20, 8.5.21, 9.5.10 (PHAR)
OS: Windows 10 64x

Output of php -m:

[PHP Modules]                       
bcmath                              
bz2                                 
calendar                            
Core                                
ctype                               
curl                                
date                                
dom                                 
exif                                
fileinfo                            
filter                              
gd                                  
gettext                             
gmp                                 
hash                                
iconv                               
imap                                
intl                                
json                                
ldap                                
libxml                              
mbstring                            
mysqli                              
mysqlnd                             
openssl                             
pcre                                
PDO                                 
pdo_mysql                           
pdo_sqlite                          
Phar                                
readline                            
Reflection                          
session                             
SimpleXML                           
soap                                
sockets                             
SPL                                 
standard                            
tokenizer                           
xml                                 
xmlreader                           
xmlrpc                              
xmlwriter                           
xsl                                 
zip                                 
zlib                                
                                    
[Zend Modules]                     

[costdev]

Command: phpunit -c phpunit.xml.dist --filter Tests_Embed_Template
PHP: 7.4.24
PHPUnit: 9.5.10 (Used PHAR to see if that made any difference on my end)
Results: Tests pass

Diff output of php -m

- Modules only you have
- bcmatch
- bz2
- gmp
- imap
- ldap
- pdo_sqlite
- soap
- xmlrpc

+ Modules only I have
+ FFI
+ ftp
+ imagick
+ pcntl
+ posix
+ shmop
+ sodium
+ sysvmsg
+ sysvsem
+ sysvshm
+ xdebug
+ Zend OPcache