Performance tests failing due to 404ing test data.

[peterwilsoncc]

The performance tests make use of the theme test data stored in the GitHub repo ​WPTT/theme-test-data/.

For some reason this repo has gone missing which is causing the build to fail. I'm not sure if the repo has been switched to private or deleted.

I've asked for additional info in the #themereview slack channel but until this is resolved it would be good to consider some alternatives:

• using an different source
• storing the theme data in WordPress-Develop to avoid third party dependencies.
• using WP CLI's generate commands to populate the data

The WPTT/

This PR adds the themeunittestdata.wordpress.xml directly to the wordpress-develop repository in tests/performance/data/ to remove the reliance on a third-party dependency.

The contents of themeunittestdata.wordpress.xml were taken from ​the Wayback Machine snapshot of the file on April 3rd, 2023.

[costdev]

​PR 4636 adds the themeunittestdata.wordpress.xml file directly to the repository to remove the reliance on an "external" dependency.

If this file becomes available again and an "external" dependency is preferable, this change could be reversed at this time.

[kafleg]

Thank you for creating the ticket. We are already checking this and will try to resolve this ASAP.

[peterwilsoncc]

Provided the tests run just dandy, I'll commit the ​linked pull request as a temporary fix.

With beta approaching, failing tests will add a lot of noise that contributors could do without.

[peterwilsoncc]

The ​WPTT/theme-test-data repo appears to have been restored so going to leave well-enough alone as the tests are now passing.

I'll leave this ticket open as it doesn't seem like a bad idea to remove the external dependency using the approach in the linked pull request but I'll leave that decision up to the perf team.

[mukesh27]

Replying to peterwilsoncc:

The ​WPTT/theme-test-data repo appears to have been restored so going to leave well-enough alone as the tests are now passing.

I'll leave this ticket open as it doesn't seem like a bad idea to remove the external dependency using the approach in the linked pull request but I'll leave that decision up to the perf team.

Agreed with @peterwilsoncc

[mukesh27]

Let's remove has-unit-tests and add dev-feedback

@mukeshpanchal27 I agree that we should wait for input from other Performance and Themes team members. This was a stopgap fix to get RPs/commits working in the short term, and potentially to remove the reliance on a dependency outside the wordpress-develop repo in the long term if that's desired.

Regarding overhead, I guess that would partially depend on how often the test data tends to get updated. I'm not aware of the frequency of this. That said, I appreciate the benefits of having a file maintained elsewhere that can receive faster updates without PR authors needing to merge/rebase on trunk.

Everyone piled on to get it resolved as quickly as possible, and this PR was one of those solutions (a stopgap) in case the repo was unavailable for an extended period of time.

We don't yet know exactly what caused the repo to become unavailable, so we may consider this a blip, or we may consider whether something should change so that the test suite is more stable.

[spacedmonkey]

CC @joemcgill

[jrf]

I wonder if as an alternative, it should be considered to move that repository to the WordPress GitHub organisation to ensure it is within the control of WP Core which clearly has a hard dependency on it... ?

/cc @dd32

The test data gets updated about once every two years. Ideally, something other than one big XML file would be used.
Like a CLI command to install different types of data, not everyone is interested in testing "classic" content, or old block markup.

[joemcgill]

I reviewed and approved @costdev's PR, should we decide to copy this file into the Core repo. Given that the repository issues have been resolved and there are active conversations going to move that repo under control of the WordPress org, I'd prefer to hold off on making this change for now to avoid having to update this file in two places in the future and to help promote the maintenance of the theme unit test project.

Account for relocated repository

https://core.trac.wordpress.org/ticket/58569

[peterwilsoncc]

The repository has been relocated to the WordPress org on GitHub.

I've created a PR to update the URL the theme test suite is downloaded from and will commit that once everything is passing. I'll close this ticket out as fixed upon the commit.

[peterwilsoncc]

In 55976:

Build/Test Tools: Update URL of theme unit test data.

Update the performance tests to account for the relocation of the theme unit test data repository to the WordPress organisation on GitHub.

Props costdev, kafleg, williampatton, dd32, otto42, poena, jrf, joemcgill, peterwilsoncc, mukesh27.
Fixes #58569.

Committed r55976 / ​https://github.com/WordPress/wordpress-develop/commit/b638c200148411007b7eb7002e996d7705a25c06

Superseded by ​https://github.com/WordPress/wordpress-develop/pull/4668