Multiple runs of the Test Old Branches workflow in a short period of time can cause a stampede of automatic reruns and Slack notifications

[desrosj]

GitHub Actions workflows are configured to automatically rerun when the first run attempt is cancelled or fails. They are also configured so that when the triggering event for a run is not pull_request the concurrency.group value is WORKFLOWNAME-SHA. Because old branches do not receive updates often, the runs triggered by workflow_dispatch usually have the same concurrency value for m any consecutive runs.

When the Test Old Branches workflow runs more than once in short period of time, this causes a cascade of cancellations, re-runs, and Slack notifications. Consider the following example:

• Commit made to trunk that runs Test Old Branches workflow.
• Run 1 starts
• Second commit made to trunk that runs Test Old Branches workflow.
• Run 2 starts
• Incomplete runs triggered by the first commit are cancelled based on the concurrency group
• Run 1 is restarted
• Incomplete runs triggered by the second commit are cancelled based on the concurrency group
• Run 2 is restarted
• Incomplete runs triggered by the first commit are cancelled for the second time based on the concurrency group
• Run 1 cancellations send a Slack notification to raise awareness.

This is then made worse because a successful run following a cancelled or failed run is considered "fixed", which is also a scenario where a Slack notification is pushed.

With 25+ branches eligible to receive security updates as a courtesy, this means that roughly 150 workflow runs are triggered every time the Test Old Branches workflow runs (using 25 branches * 6 workflows each, which is a rough average) and potentially roughly the same number of notifications for cancellations, then the same number for "fixes". This can be even worse if the commit made in trunk caused legitimate failures or also experienced timeouts/cancellations, and when a number of other commits are made around the same time.

The cancelled run Slack notification is configured to only send a notice for only the second run attempt to avoid needlessly sharing a timeout that resolves itself during the automatic re-run (see r62740). However, the cancelled outcome in GitHub Actions represents a few different outcomes that are not easy to clarify. A cancellation could be a timeout, a user initiated cancellation, a concurrency-based cancellation, and sometimes a platform-level problem can cause cancellations.

Because of this there's no logic included to determine if the cancellation was caused by the configured concurrency groupings, which is an expected and desired outcome and does not warrant a rerun or Slack notification.

​See this Slack history for an example of how this materializes in #core.

While the REST API documents timed_out as a possible value for a job's conclusion, it turns out that value is never surfaced for a job object.

However, a cancellation and a time out can be determined by examining the annotations associated with a given job through the Checks API.

The one trade off of this approach is that checks: read must be added to all workflows that call this reusable one. Permissions are audited at the beginning of a run and it fails immediately when necessary permissions are lacking. So that would need to be backported to older branches right away.

If it's determined this is not worth the effort, then the original r63310 and r63313 commits should be reverted and the cancelled workflow notification can be adjusted to note that cancellations may also include time outs.

Trac ticket: Core-66048

## Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Used for: Full investigation, implementation, and live validation. Manually reviewed by myself and heavily modified the inline comments.

[desrosj]

Some ideal outcomes. When a workflow is cancelled because of the configured concurrency.group:

• it should not be re-run automatically.
• it should be ignored by subsequent runs when looking up the status of the previous run to see if a failure or undesired cancellation (timeout, manual cancellation) was "fixed".

Also, it would be great to post a distinct Slack notification when a timeout occurs vs. a cancellation. This was attempted in r3310, but that approach did not work as expected. I ​have a PR with a different approach using the Checks API. I've changed the PR description to point to this ticket instead of the more general #65845. I was initially hesitant to add code that requires a new permission (which requires changes to the workflow files in all old branches). But given that it would also help with fixing this issue, I think that approach has become a lot more reasonable.