Make WordPress Core

Opened 8 months ago

Closed 8 months ago

Last modified 8 months ago

#58569 closed defect (bug) (fixed)

Performance tests failing due to 404ing test data.

Reported by: peterwilsoncc's profile peterwilsoncc Owned by: peterwilsoncc's profile peterwilsoncc
Milestone: 6.3 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch dev-feedback
Focuses: performance Cc:

Description

The performance tests make use of the theme test data stored in the GitHub repo WPTT/theme-test-data/.

For some reason this repo has gone missing which is causing the build to fail. I'm not sure if the repo has been switched to private or deleted.

I've asked for additional info in the #themereview slack channel but until this is resolved it would be good to consider some alternatives:

  • using an different source
  • storing the theme data in WordPress-Develop to avoid third party dependencies.
  • using WP CLI's generate commands to populate the data

Change History (17)

This ticket was mentioned in PR #4636 on WordPress/wordpress-develop by @costdev.


8 months ago
#1

  • Keywords has-patch has-unit-tests added

The WPTT/

This PR adds the themeunittestdata.wordpress.xml directly to the wordpress-develop repository in tests/performance/data/ to remove the reliance on a third-party dependency.

The contents of themeunittestdata.wordpress.xml were taken from the Wayback Machine snapshot of the file on April 3rd, 2023.

#2 @costdev
8 months ago

PR 4636 adds the themeunittestdata.wordpress.xml file directly to the repository to remove the reliance on an "external" dependency.

If this file becomes available again and an "external" dependency is preferable, this change could be reversed at this time.

Last edited 8 months ago by costdev (previous) (diff)

#3 @kafleg
8 months ago

Thank you for creating the ticket. We are already checking this and will try to resolve this ASAP.

Version 0, edited 8 months ago by kafleg (next)

#4 @peterwilsoncc
8 months ago

Provided the tests run just dandy, I'll commit the linked pull request as a temporary fix.

With beta approaching, failing tests will add a lot of noise that contributors could do without.

#5 follow-up: @peterwilsoncc
8 months ago

The WPTT/theme-test-data repo appears to have been restored so going to leave well-enough alone as the tests are now passing.

I'll leave this ticket open as it doesn't seem like a bad idea to remove the external dependency using the approach in the linked pull request but I'll leave that decision up to the perf team.

#6 in reply to: ↑ 5 @mukesh27
8 months ago

Replying to peterwilsoncc:

The WPTT/theme-test-data repo appears to have been restored so going to leave well-enough alone as the tests are now passing.

I'll leave this ticket open as it doesn't seem like a bad idea to remove the external dependency using the approach in the linked pull request but I'll leave that decision up to the perf team.

Agreed with @peterwilsoncc

#7 @mukesh27
8 months ago

  • Keywords dev-feedback added; has-unit-tests removed

Let's remove has-unit-tests and add dev-feedback

@costdev commented on PR #4636:


8 months ago
#8

@mukeshpanchal27 I agree that we should wait for input from other Performance and Themes team members. This was a stopgap fix to get RPs/commits working in the short term, and potentially to remove the reliance on a dependency outside the wordpress-develop repo in the long term if that's desired.

Regarding overhead, I guess that would partially depend on how often the test data tends to get updated. I'm not aware of the frequency of this. That said, I appreciate the benefits of having a file maintained elsewhere that can receive faster updates without PR authors needing to merge/rebase on trunk.

Everyone piled on to get it resolved as quickly as possible, and this PR was one of those solutions (a stopgap) in case the repo was unavailable for an extended period of time.

We don't yet know exactly what caused the repo to become unavailable, so we may consider this a blip, or we may consider whether something should change so that the test suite is more stable.

#9 @spacedmonkey
8 months ago

CC @joemcgill

#10 @jrf
8 months ago

I wonder if as an alternative, it should be considered to move that repository to the WordPress GitHub organisation to ensure it is within the control of WP Core which clearly has a hard dependency on it... ?

/cc @dd32

@poena commented on PR #4636:


8 months ago
#11

The test data gets updated about once every two years. Ideally, something other than one big XML file would be used.
Like a CLI command to install different types of data, not everyone is interested in testing "classic" content, or old block markup.

#12 @joemcgill
8 months ago

I reviewed and approved @costdev's PR, should we decide to copy this file into the Core repo. Given that the repository issues have been resolved and there are active conversations going to move that repo under control of the WordPress org, I'd prefer to hold off on making this change for now to avoid having to update this file in two places in the future and to help promote the maintenance of the theme unit test project.

This ticket was mentioned in PR #4668 on WordPress/wordpress-develop by @peterwilsoncc.


8 months ago
#13

Account for relocated repository

https://core.trac.wordpress.org/ticket/58569

#14 @peterwilsoncc
8 months ago

The repository has been relocated to the WordPress org on GitHub.

I've created a PR to update the URL the theme test suite is downloaded from and will commit that once everything is passing. I'll close this ticket out as fixed upon the commit.

#15 @peterwilsoncc
8 months ago

  • Owner set to peterwilsoncc
  • Resolution set to fixed
  • Status changed from new to closed

In 55976:

Build/Test Tools: Update URL of theme unit test data.

Update the performance tests to account for the relocation of the theme unit test data repository to the WordPress organisation on GitHub.

Props costdev, kafleg, williampatton, dd32, otto42, poena, jrf, joemcgill, peterwilsoncc, mukesh27.
Fixes #58569.

Note: See TracTickets for help on using tickets.