Make WordPress Core

Opened 7 months ago

Last modified 6 weeks ago

#59416 new task (blessed)

Add a GitHub Action which alerts contributors to a WordPress Playground link to use for testing PRs

Reported by: jeffpaul's profile JeffPaul Owned by:
Milestone: 6.6 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch
Focuses: Cc:


The following approach could help bring in many new folks to help with testing PRs as they no longer need to have a local set up and able to load patches/PRs locally and can quickly jump into a WordPress Playground environment to help test.

Gutenberg has a GitHub Action that builds a ZIP of the plugin (workflow file, example Gutenberg PR with built ZIP attached) which then WordPress Playground makes use of in the Gutenberg PR Previewer.

Taking that above functionality a step further and integrating for Core:

  1. Add a GitHub Action for wordpress-develop that builds a ZIP of core for a PR (similar to the Gutenberg action).
  2. Have that same action also add a comment to the PR with a link to a WordPress Playground that's built using the ZIP from the PR. So a comment like:

    Thanks <@PR-author>! Here's a link to a site created from this PR that can be used for testing: <wordpress-playground-link>.

In my quick review of the WordPress Playground documentation it does reference an ability to pull requests from your repository, but I think that's crafted more for plugin developers and not for WordPress core itself. I took a look at ReplaceSiteStep, but I think that's just for loading in alternate content and not a full on separate WordPress ZIP. So, this functionality might require additional functionality within WordPress Playground but I'll try and chase down folks from that team to comment here on the topic.

Change History (60)

#1 @dmsnell
7 months ago

This sounds great, @JeffPaul.

One reason that the Playground defaults to official WordPress images is because it takes some work to strip out content that isn't needed and prevent downloading a large initial image to boot up. If you wanted to build a step to load a PR though I think that would be awesome. People wanting to test PRs against wordpress-develop would understand that there could be more initial download, and they are likely to have higher-than-average download speeds.

Let me know how I can help; I'm mostly listed to one day a week on the Playground and am not the best guide, but I can try.

#2 @antoniosejas
7 months ago

Great idea! Thanks for sharing all the links.
I'd love to help make this happen. I'll investigate if Blueprints are enough to accomplish what we want.

#3 @JeffPaul
6 months ago

Note that and help advance this topic. Review/feedback/help on those PRs would be wonderful for anyone interested.

This ticket was mentioned in PR #5481 on WordPress/wordpress-develop by @zieladam.

6 months ago

  • Keywords has-patch added

Creates a new GitHub workflow to expose WordPress build as a GitHub artifact. The goal is to enable previewing Pull Requests in WordPress Playground.

Related Playground PR

Trac Ticket:

cc @gziolo @youknowriad @peterwilsoncc @desrosj

@Bernhard Reiter commented on PR #5481:

6 months ago

And I just added a zip -r build line that sped up the upload step from 10 minutes to 19s. Woah!

That's due to a somewhat familiar issue with the upload-artifact action: Uploading multiple files takes significantly more time than uploading a single file 😕

One thing to keep in mind here is that this results in the zip file that is uploaded as an artifact being zipped again. It's not immediately apparent, but it may require an additional step when using the artifact.

FWIW, the same tradeoff applies to Gutenberg (where it has confused users that uploaded the "double-zipped" to their WP, expecting it to work like a "regular" plugin zip file); there's also an issue in the `upload-artifact` repo about the double-zip. We even attempted removing that extraneous zipping at some point ( but concluded that it wasn't worth the negative performance impact.

@zieladam commented on PR #5481:

6 months ago

Shall we wait for 6.4 to be branched from trunk before landing this?

This change matters for trunk Pull Requests and even if the workflow file changes, I don't think we'll need to backport that. I'll go ahead and merge. Feel free to revert if I'm wrong here.

#7 @zieladam
6 months ago

In 56958:

Build/Test Tools: Store for every GitHub Pull Request as a GitHub artifact.

Storing build files enables reusing them in WordPress Playground and ultimately implementing a Pull Request.

Props desrosj, bernhard-reiter
See #59416.

#9 @zieladam
6 months ago

I just shipped wordpress-develop PR previewer!

You can even link directly to PRs:

Two notes:

  • If your PR doesn’t work, rebase it. It needs to target the latest version of trunk with this commit in it.
  • Downloading the built PR can take a while. Let’s find a way to optimize that!

Other than that, everything seems ripe to create a GitHub action which alerts contributors to a WordPress Playground link to use for testing PRs

#10 @jorbin
6 months ago

  • Milestone changed from Awaiting Review to 6.4

To solve the double zip issue, what if the cd build && zip -r . step was replaced with rm -r $(ls -A | grep -v build ) && mv build/* . && rmdir build which moves everything into the main folder?

I also updated the Milestone since this has a commit in 6.4.

#11 @zieladam
6 months ago

Thanks @jorbin! Double zipping is an unfortunate side-effect of how GitHub processes artifacts. If you upload a directory with multiple files, GitHub will process them one by one, take a long time to do it, and eventually zip everything into a single bundle. If you upload a zip file, it will be very fast but GitHub will zip it again anyway. It's not really an issue, just a small annoyance.

This ticket was mentioned in PR #5526 on WordPress/wordpress-develop by @zieladam.

6 months ago

Very simple GitHub workflow to comment on every trunk PR with a link to a Playground-based preview.

Other than access, it seems to be fine. The last thing to solve is the 403 error:

403 Resource not accessible by integration

cc @jeffpaul

Trac Ticket: 59416

#13 @zieladam
6 months ago

I did a PR with a simple GitHub workflow to comment on every trunk PR with a link to a Playground-based preview:

One issue with it is access – GitHub complains that workflow has no access rights to comment. I suppose we need an access token or some sort of repository setup to allow comment creation – any ideas how to address that @JeffPaul @peterwilsoncc @akirk?

@JeffPaul commented on PR #5526:

6 months ago

This action adds a comment, so perhaps there's something to glean from that to leverage here?

cc: @desrosj

#15 @desrosj
6 months ago

In 56973:

Build/Test Tools: Skip Puppeteer download in build workflow.

This adds the PUPPETEER_SKIP_DOWNLOAD environment variable to the Build WordPress workflow to skip downloading Puppeteer browser binaries unnecessarily.

Follow up to [56958].

See #59416, #59517, #58863.

@zieladam commented on PR #5526:

6 months ago

Thanks you @desrosj @swissspidy! I just addressed the feedback on this PR, although we won't able to test it without merging. GitHub actions docs says:

This event runs in the context of the base of the pull request, rather than in the context of the merge commit, as the pull_request event does. This prevents execution of unsafe code from the head of the pull request that could alter your repository or steal any secrets you use in your workflow. This event allows your workflow to do things like label or comment on pull requests from forks. Avoid using this event if you need to build or run code from the pull request.

@zieladam commented on PR #5526:

6 months ago

I don't think we should have a separate workflow for this. I think that this should be a step within the Build WordPress workflow after the artifact is built and uploaded.

This workflow now uses a different trigger (pull_request_target) than the Build WordPress workflow so it may need to remain separate.

@swissspidy commented on PR #5526:

6 months ago

@adamziel you can test this by opening a PR on your own fork instead this repository. This way we can refine it before merging.

We can also try using a different trigger with a custom access token if we want to merge the workflow files.

@desrosj commented on PR #5526:

6 months ago

What was the reason pull request wouldn't work? I think I'm missing that.

@swissspidy commented on PR #5526:

6 months ago

@desrosj correct me if I‘m wrong, but permissions are different for that trigger. See

@desrosj commented on PR #5526:

6 months ago

Ah, sorry I missed that because it was collapsed in the resolved discussion. We should be able to use permissions to fine tune the capability we need the workflow, job, or step to have.

@desrosj commented on PR #5526:

6 months ago

Sorry for all the noise!

After some testing, I think you're right @swissspidy. I could have sworn that in the past I was able to get commenting to work within a PR, but I must be mixing that up either with workflows that are _triggered_ by other workflows within a PR (and thus run within the context of the repo itself) or instances where I had a PR from a branch within the same repository and not a fork.

I still feel like we should not have a separate workflow just for commenting. The commenting is related to the build step, and even depends on it (we shouldn't comment unless we know that a build has successfully been created). I've gone and combined the two, and also renamed build.yml to wordpress.playground.yml, which will allow us to include other additional logic related to Playground in the future.

A few other notes about the changes I included:

  • I pinned a specific commit SHA to the version of actions/github-scripts. This is a recommended best practice in GitHub's hardening recommendations when using third-party actions.
  • I've made the commenting job dependent on the build one. We shouldn't comment on the PR until there actually is an artifact that can be used.
  • I removed the opened type filter for the event as well. This ensures that the build runs on every update to the PR.
  • To prevent comments being posted every time the workflow runs, I've changed the commenting job to look for a preexisting comment before leaving a new one. This also solves the edge case where the first build attempt for a PR fails. The comment job will skip the first time, and it will run once there is a successful build.
  • I expanded the comment a bit to call out some of the limitations and "gotchas" about using a Playground instance. I also linked off to the Limitations page in the documentation so someone can dig deeper if they would like.

One other thought. I'm not sure that running on workflow_dispatch is useful if a Playground instance cannot be created using a commit hash. Something like Even if that's possible, I think that adding a push trigger event would be better. That would ensure every commit to trunk (or whatever branches we specify) has a built artifact that could be used in a Playground instance.

@desrosj commented on PR #5526:

6 months ago

Also, here is an example of what the content of the comment will look like:

## Test using WordPress Playground
The changes in this pull request can previewed and tested using a WordPress Playground instance.

WordPress Playground is an experimental project that creates a full WordPress instance entirely within the browser.

### Some things to be aware of

  • The Plugin and Theme Directories cannot be accessed within Playground.
  • All changes will be lost when closing a tab with a Playground instance.
  • All changes will be lost when refreshing the page.
  • A fresh instance is created each time the link below is clicked.
  • Every time this pull request is updated, a new ZIP file containing all changes is created. If changes are not reflected in the Playground instance, it's possible that the most recent build failed, or has not completed. Check the list of workflow runs to be sure.

For more details about these limitations and more, check out the Limitations page in the WordPress Playground documentation.

Test this pull request WordPress Playground.

@swissspidy commented on PR #5526:

6 months ago

I've made the commenting job dependent on the build one. We shouldn't comment on the PR until there actually is an artifact that can be used. This logic is untested though since we can't verify until merged.

And why can't we test this by opening a PR against the fork? That's how I successfully tested this in the past:

@zieladam commented on PR #5526:

6 months ago

I'm on a meetup this week and AFK the next one – feel absolutely free to take over here

#26 @jorbin
6 months ago

  • Milestone changed from 6.4 to 6.5

Moving to milestone 6.5 as WP 6.4 RC3 has been released.

@zieladam commented on PR #5526:

5 months ago

@desrosj did you manage to get it to work in your fork?

@desrosj commented on PR #5526:

5 months ago

I did, but I have a bit of cleanup to do. I plan to circle back to this later this week after 6.4 is out!

@desrosj commented on PR #5526:

5 months ago

OK, I think I've cleaned up my work and have it ready to go. You can see a test comment in desrosj/wordpress-develop#141.

I chatted with @swissspidy, and thought about trying this way forward. We are already building WordPress in the workflow that tests the build process. So it makes sense to just perform the zipping and uploading of the artifact in that workflow instead of having a completely different one. That doesn't solve the issue with requiring pull_request_target though.

To solve this, I've consolidated the welcome comment workflow with the logic that comments with Playground details. It will run when a PR is opened on pull_request_target for the welcome message, but it will only post the comment with the Playground details after the build workflow has succeeded. This ensures that there is an artifact present that can actually be used. This is triggered using the workflow_dispatch event, which does not have the same issue as the pull_request event.

I don't love that it will run every time a PR is updated, but I'm not sure any way around that.

@zieladam commented on PR #5526:

5 months ago

@desrosj I just reviewed and this looks good to me, thank you so much for taking the lead here! It indeed a pity this needs to run on every PR update, but I don't think it's a big deal. I like how you consolidated that with the new contributor comment 🎉 Let's ship it!

#31 @desrosj
5 months ago

In 57124:

Build/Test Tools: Comment on PRs with WordPress Playground information.

After [56958], pull requests to wordpress-develop can be tested within a WordPress Playground instance. However this capability is not obvious.

This makes several updates to GitHub action workflows to avoid unnecessary extra runs/jobs while also commenting on pull requests with proper context for testing in Playground once a build is available for the PR.


  • The build.yml workflow has been deleted. The WordPress build process is already being tested within the test-build-processes.yml workflow. Now that workflow will produce and upload the built artifact for the given PR.
  • A pull-request-comments.yml workflow has been added to handle the commenting. To avoid having multiple workflows that comment on PRs, the logic within the welcome-new-contributors.yml workflow has been merged into this one.
  • The pull_request event trigger lacks the needed permission to comment on or update pull requests. Instead of using pull_request_target to navigate around this, the test-build-processes.yml workflow will now use workflow_dispatch to initiate a run of the commenting workflow. This adds the ability to only comment on the PR when a build has been successfully created and uploaded as an artifact.

Props swissspidy, zieladam, jeffpaul.
See #59416.

#33 @desrosj
5 months ago

In 57125:

Build/Test Tools: Allow the playground-comment job to fail.

When this job fails, it should not fail the entire workflow. The next update to the pull request will attempt to dispatch the commenting workflow.

Follow up to [57124].

Props johnbillion.
See #59416.

@zieladam commented on PR #5526:

5 months ago

@desrosj weird! Someone on StackOverflow suggested this:

go to and in Workflow Permissions section give actions Read and Write permissions. That provides your token with rights to modify your repo and solves your problem.

@zieladam commented on PR #5526:

5 months ago

@desrosj interestingly, the "first contributor" comment still works:

@zieladam commented on PR #5526:

5 months ago

So workflow_dispatch doesn't come with the same note about permissions as `pull_request_target`:

For workflows that are triggered by the pull_request_target event, the GITHUB_TOKEN is granted read/write repository permission

What if we just lean on pull_request_target and add a message on WordPress Playground side like "this Pull Request is still being built – please wait a few moments"?

@zieladam commented on PR #5526:

5 months ago

I found one more issue with the uploaded artifact – it's 332 bytes and doesn't actually contain the built files:

This ticket was mentioned in PR #5737 on WordPress/wordpress-develop by @zieladam.

4 months ago

Follows up on

Updates the pull-request-comments.yml workflow to post the Playground comment on the pull_request_target trigger, not on the workflow_dispatch trigger.

Here's how it works in my fork:

The previous attempt leaned on workflow_dispatch. The goal was to only post a comment once the build workflow creates the artifact required by the PR previewer. However, workflow_dispatch seems to have less permissions than the pull_request_target trigger. The workflows dispatched that way fail with a 403 error and the following error message:

data: {
      message: 'Resource not accessible by integration',
      documentation_url: ''

Since there doesn't seem to be a straightforward way to only post the comment when the build artifact is ready, I propose posting it immediately and then handling the UX on the PR previewer side. It could display a progress indicator and say "That Pull Request is still being built on GitHub, you'll be redirected to the preview as soon as it's ready!"

See how the commenting workflow failed on all these PRs:

## Testing instructions

Trac Ticket:

cc @desrosj @swissspidy

This ticket was mentioned in PR #5742 on WordPress/wordpress-develop by @zieladam.

4 months ago

Moves the zipping step before cleaning the dist directory.

The callable-test-core-build-process.yml zips the WordPress dist directory *after* npm run grunt clean runs. This produces a mostly empty zip artifact that's only 332 bytes.

#40 @zieladam
4 months ago

In 57174:

Build/Test Tools: Zip WordPress files before npm run clean

Fixes a problem in WordPress artifact upload pipeline. The callable-test-core-build-process.yml used to zip the WordPress dist directory after npm run grunt clean runs, producing an empty zip file. This commit moves the zip before the cleanup task.

Follow up to [57124].

Props ockham, dmsnell.
See #59416.

@swissspidy commented on PR #5742:

4 months ago

Note that you'll need to update the order of the comments above as well.

#43 @SergeyBiryukov
4 months ago

In 57177:

Docs: Correct the order of steps in the WordPress Core build process workflow.

Follow up to [57124], [57174].

Props swissspidy.
See #59416.

@zieladam commented on PR #5737:

4 months ago

cc @ockham

#45 @zieladam
4 months ago

In 57178:

Build/Test Tools: Fix the workflow that posts a preview link on every Pull Request

Fixes a 403 error in the pull-request-comments.yml job by switching from a custom workflow dispatch call to the pull_request_target trigger.

Follow up to [57124], [57174].
Props ockham.
See #59416.

This ticket was mentioned in PR #5752 on WordPress/wordpress-develop by @zieladam.

4 months ago

This PR exists to test the user experience of previewing WordPress Pull Requests.


@zieladam commented on PR #5752:

4 months ago

Let's keep this PR open until 20.12 as a live demo of the previewer. In the meantime, other PRs will get created and rebased so there will be ample examples of the previewer.

#49 @zieladam
4 months ago

I've merged that launches the PR commenting workflow using the same mechanism as the "First contribution" workflow.

This fixes the CI checks for wordpress-develop PRs that failed due to the 403 error encountered by the previous version of that workflow.

Caveat: The comment with the preview link is now posted before the build artifact is ready, which could cause confusion if it led the user to an error page. This is why I also adjusted the user experience on WordPress Playground side of things to communicate how the build artifact is not ready yet, and that the previewer will auto-retry every 30 seconds.

Here's a PR I prepared to showcase how it works:

Let's let this ticket sit for a bit to and gather feedback before closing.

Version 0, edited 4 months ago by zieladam (next)

This ticket was mentioned in Slack in #core by jeffpaul. View the logs.

4 months ago

#51 @zieladam
4 months ago

A nice improvement would be to only post the preview link if the PR updated the src directory. It might not make much sense to preview PRs adjusting, say, GitHub workflows.

Also, noting here that @swissspidy explored using the workflow_run trigger to only post the comment once the build artifact is available to download:

#52 @Ankit K Gupta
4 months ago

I recently tried out this feature and successfully tested a core ticket using the Playground.
It's truly amazing and has proven to be an excellent tool so far. The ease of use and efficiency make it incredibly helpful, saving a significant amount of time in the testing process.

Thanks, everyone for this great help!

#53 @swissspidy
4 months ago

In 57210:

Build/Test Tools: Post message for testing on Playground only after build succeeds.

Uses the workflow_run trigger to only leave pull request comments after the build jobs finish.

Props zieladam, desrosj.
See #59416.

#55 follow-up: @peterwilsoncc
3 months ago

@zieladam @desrosj For the PR comments, what do you think about running the install with WP_DEBUG set to true so debug messages show?

For example instead of linking to the link would become$schema%22:%22,%22steps%22:%5B%7B%22step%22:%22defineWpConfigConsts%22,%22consts%22:%7B%22WP_DEBUG%22:true%7D%7D%5D%7D

We could include two links if you'd prefer to give people a choice.

#56 in reply to: ↑ 55 @desrosj
3 months ago

Replying to peterwilsoncc:

@zieladam @desrosj For the PR comments, what do you think about running the install with WP_DEBUG set to true so debug messages show?

I think that providing a few links to specific configurations is a great idea.

Another thing that has been on my mind is that the comment should only be left when the following things are changed:

  • The src directory
  • The (package|package-lock).json files
  • Gruntfile.js
  • tools/webpack
  • webpack.config.js

If things like GitHub workflows or test files are the only thing changed, then there's no reason to confuse a contributor since they won't actually be ble to test within the tool.

#57 @swissspidy
2 months ago

This could probably be done by analyzing the file changes in the test-build-processes.yml workflow and not upload the pr-number artifact if it's not worthwhile. Then in pull-request-comments.yml we can just silently bail if there's no artifact (rather than calling core.setFailed( 'No artifact found!' ); and causing the step to fail)

#58 @swissspidy
2 months ago

  • Type changed from enhancement to task (blessed)

#59 @swissspidy
7 weeks ago

Another thing that has been on my mind is that the comment should only be left when the following things are changed

I'd suggest opening a new ticket for that & closing this one. WDYT?

#60 @swissspidy
6 weeks ago

  • Milestone changed from 6.5 to 6.6
Note: See TracTickets for help on using tickets.