Make WordPress Core

Opened 21 months ago

Closed 20 months ago

Last modified 15 months ago

#56407 closed enhancement (fixed)

Rerun GitHub Action workflows on the first failure

Reported by: desrosj's profile desrosj Owned by: desrosj's profile desrosj
Milestone: 6.1 Priority: normal
Severity: normal Version:
Component: Build/Test Tools Keywords: has-patch commit
Focuses: Cc:

Description

It's fairly common for GitHub Action workflows to fail because of several different types of errors. Some examples include:

  • Chromium timeout errors
  • WordPress.org hiccups when making requests within PHP unit tests
  • NPM install errors
  • Docker container registry timout and connection issues.

In almost all cases (except when there are service level outages), rerunning the workflow will result in a successful outcome. Because this is currently manual, it's sometimes forgotten.

Our Actions workflows should attempt to fix these runs by automatically rerunning failed jobs.

Attachments (1)

56407.diff (3.9 KB) - added by desrosj 21 months ago.

Download all attachments as: .zip

Change History (32)

This ticket was mentioned in PR #3111 on WordPress/wordpress-develop by desrosj.


21 months ago
#1

This workflow will restart a failed or cancelled workflow when it has only been run once.

This is dependent on actions/github-script#283, which will add the needed .rest.actions.reRunWorkflowFailedJobs() function added in octokit/plugin-rest-endpoint-methods.js@v5.14.0.

Trac ticket: https://core.trac.wordpress.org/ticket/56407

desrosj commented on PR #3111:


21 months ago
#2

There is some inconsistency, where github.rest.actions.createWorkflowDispatch is sometimes called with const result = await and sometimes just called on it's own. I think one should be picked and used throughout

Ah, good spot. This was a relic from when I was debugging why the workflow_dispatch call was not working in my fork.

One thing that is not clear to me reading through the action's documentation is whether github-script will cause a step or job to fail if a non 2xx response code is returned by an API call. If not, using await and checking the response for a problem manually would be preferred. But for now I have removed the one instance of that in the coding standards workflow.

#3 @desrosj
21 months ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 53947:

Build/Test Tools: Automatically rerun a workflow the first time it fails.

There are several common reoccurring issues that sometimes cause GitHub Action workflows to fail (connection timeouts to WordPress.org or the Docker container registry, npm install failures, Chromium issues, etc.). Except when there are service level outages, most of these issues can be resolved by simply rerunning the workflow.

This introduces a new step within each of Core’s GitHub Action workflows that attempts to rerun the failed jobs within the workflow that encountered a failure if they are running for the first time. Since a workflow is not allowed to restart itself, a new failed-workflow.yml callable workflow is being introduced.

Other related adjustments in this changeset:

  • The actions/github-script 3rd-party action is also now updated to the latest version (v6.2.0).
  • A new secret, GHA_WORKFLOW_DISPATCH, has been introduced. This will replace the current one in use (GHA_OLD_BRANCH_DISPATCH) with a less specific name.

Props jorbin, desrosj.
Fixes #56407.

#5 @desrosj
21 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

The workflow itself works when manually called and providing a cancelled or failed workflow run ID (see this test), but for some reason the step is being skipped in other workflows. It worked in the test repository I made on GitHub, so not sure what is missing.

Will dig in.

@desrosj
21 months ago

#6 @desrosj
21 months ago

  • Keywords commit added

Turns out to be an incorrect variable being used. github.run_number (the unique number for a run represents for a workflow over time) is not the same as github.run_attempt (the run attempt number for a given workflow run). The attempt is the correct one here.

#7 @desrosj
21 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 54039:

Build/Test Tools: Correct the context variable being used when auto-retrying failed and cancelled workflows.

The correct variable here is github.run_attempt, which represents the unique number for each attempt of a particular workflow run in a repository.

The github.run_number currently being used represents the unique number for each run of a particular workflow in a repository.

Follow up to [53947].

Fixes #56407.

#8 @TobiasBg
21 months ago

Hi @desrosj, thanks for your work on this! Cool idea!

To have it documented: Why is the GHA_WORKFLOW_DISPATCH secret needed over e.g. the default GITHUB_TOKEN, and what permission scope does it need? Thanks!

#9 @desrosj
21 months ago

Great question, @TobiasBg!

I definitely wanted to just use GITHUB_TOKEN instead. But in my testing, GITHUB_TOKEN I found that does not have the required permissions to modify workflow runs through the REST API.

The documentation for creating a workflow dispatch event mentions actions:write is required, but that specifically mentions GitHub Apps.

GitHub Actions does support specifying custom permissions in a workflow through permissions at the top or job level, but in my testing, even specifying permissions: write-all had no effect.

It can also be a bit hard to get to the bottom of because calling github.rest.actions.createWorkflowDispatch() with a token lacking the required permissions still returns a 204 status (the one documented as expected). But eventually, I found the right documentation on this here.

When you use the repository's GITHUB_TOKEN to perform tasks, events triggered by the GITHUB_TOKEN will not create a new workflow run. This prevents you from accidentally creating recursive workflow runs... If you do want to trigger a workflow from within a workflow run, you can use a personal access token instead of GITHUB_TOKEN to trigger events that require a token.

So seems that it's an intentional design decision to prevent user error.

#10 @desrosj
21 months ago

This came through today: https://github.blog/changelog/2022-09-08-github-actions-use-github_token-with-workflow_dispatch-and-repository_dispatch/. Looks like using the supplied token is now allowed! I’ll update the workflows through #55652.

#11 @desrosj
20 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopening to add the enable_debug_logging parameter when restarting a workflow. See https://docs.github.com/en/rest/actions/workflow-runs#re-run-failed-jobs-from-a-workflow-run.

#12 @desrosj
20 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 54264:

Build/Test Tools: Enable debugging when rerunning a failed workflow.

Follow up to [53947], [54039].

Fixes #56407.

#13 @desrosj
15 months ago

In 55516:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.

Merges [53736], [53737], [53940], [53947], [54039], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.9 branch.
See #55652, #56407, #54695, #56820, #56816, #56793, #56820, #57572.

#14 @desrosj
15 months ago

In 55517:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.8 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#15 @desrosj
15 months ago

In 55518:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.8 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#16 @desrosj
15 months ago

In 55519:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.6 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#17 @desrosj
15 months ago

In 55520:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.5 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#18 @desrosj
15 months ago

In 55521:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.4 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#19 @desrosj
15 months ago

In 55522:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.3 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#20 @desrosj
15 months ago

In 55523:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54649], [54650], [54651], [54674], [54750], [54852], [55152], [55487] to the 5.2 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#21 @desrosj
15 months ago

In 55524:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 5.1 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#22 @desrosj
15 months ago

In 55525:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 5.0 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#23 @desrosj
15 months ago

In 55527:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.9 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#24 @desrosj
15 months ago

In 55528:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.8 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#25 @desrosj
15 months ago

In 55529:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.7 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#26 @desrosj
15 months ago

In 55530:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.6 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#27 @desrosj
15 months ago

In 55531:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.5 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#28 @desrosj
15 months ago

In 55532:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.4 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#29 @desrosj
15 months ago

In 55533:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.3 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#30 @desrosj
15 months ago

In 55534:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.2 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

#31 @desrosj
15 months ago

In 55535:

Build/Test Tools: Backport updates to GitHub Actions.

This backports several changesets to GitHub Actions workflows. These changesets:

  • address the deprecated notices related to save-output and set-output to ensure the workflows continue to run after these are removed.
  • adds support for automatically retrying a failed workflow once.
  • removes workflow files that are not applicable to the branch.
  • backports some Docker environment related tooling updates for the sake of consistency across branches.

Merges [53736], [53737], [53940], [53947], [54039], [54096], [54108], [54293], [54313], [54342], [54343], [54373], [54511], [54650], [54651], [54674], [54750], [54852], [55152], [54651], [55487] to the 4.1 branch.
See #55652, #56407, #56528, #54695, #56820, #56816, #56793, #56820, #57572.

Note: See TracTickets for help on using tickets.