Automatically cancel pending Travis builds with each commit

[johnbillion]

This might be a meta ticket or it might be a @jorbin ticket.

When many commits land in core within a short (or not so short) period of time, a backlog of ​pending builds can build up on Travis, which means committers have to wait longer and longer to see the result of their commit. Sometimes interim builds are manually cancelled by users that have been given access by @jorbin.

Ideally, any pending builds (builds that have yet to start) would be automatically cancelled when each new build gets triggered by a commit. That way, when a build completes, all interim builds will be skipped and the most recent build will begin.

This works well for the WordPress project because we only have a single branch and don't need to worry about builds for other branches.

Travis has ​an API that could be used to cancel pending builds when each new build arrives.

Thoughts?

[jeremyfelt]

One reason it's nice to have a full build for each commit is so that we know when a failing test was introduced. If we cancelled pending jobs, we'd be okay most of the time. Every once and a while we may find ourselves tracking things down.

Other thoughts:

• Can we set our tests to stop on failure instead of continuing to process the remainder?
• Is there a way we can cancel remaining jobs inside a build if more than 1 job fails?
• Can we reduce the number of PHP versions we test against? How often has something failed on 5.3/4/5 and passed on 5.2/6 and 7?

Of course, things run mostly concurrent right now, so some of those may not help at all. Especially when the biggest slow down is waiting for everything to pass.

Interesting reading: ​https://www.drupal.org/drupalorg/blog/drupalci-continuous-integration-testing-for-drupalorg

We are kind of stretching the bounds of free Travis server power.

This is probably a meta topic. :)

[netweb]

Replying to johnbillion:

Sometimes interim builds are manually cancelled by users that have been given access by @jorbin.

I think all committers should have access to cancel and restart Travis CI jobs.

Ideally, any pending builds (builds that have yet to start) would be automatically cancelled when each new build gets triggered by a commit. That way, when a build completes, all interim builds will be skipped and the most recent build will begin.

This works well for the WordPress project because we only have a single branch and don't need to worry about builds for other branches.

Travis has ​an API that could be used to cancel pending builds when each new build arrives.

I've created a ticket asking Travis CI to support "Rolling Builds" ​https://github.com/travis-ci/travis-ci/issues/6782

AppVeyor, the Windows based CI has this feature:

​https://www.appveyor.com/docs/build-configuration/#rolling-builds
“Rolling builds” are great for very active OSS projects with lengthy queue. Whenever you do a new commit to the same branch OR pull request all current queued/running builds for that branch or PR are cancelled and the new one is queued. Other words, rolling builds make sure that only the most recent commit is built.

Replying to jeremyfelt:

One reason it's nice to have a full build for each commit is so that we know when a failing test was introduced. If we cancelled pending jobs, we'd be okay most of the time. Every once and a while we may find ourselves tracking things down.

Agree, "most of the time" we'd be okay.

Other thoughts:

• Can we set our tests to stop on failure instead of continuing to process the remainder?
• Is there a way we can cancel remaining jobs inside a build if more than 1 job fails?

We used to be able to perform the above somewhat, we had to remove this in [35150] due to a ​Travis CI bug.

What used to happen before [35150] was a notice would be piped into the Slack #core channel as soon as a single job would fail, the remaining jobs would then continue to run as reporting against the other PHP versions was also worth knowing. Subsequent build jobs after that first failure could be cancelled by users with permissions to restart or cancel jobs on Travis CI, this can be done per job or per build as needed.

• Can we reduce the number of PHP versions we test against? How often has something failed on 5.3/4/5 and passed on 5.2/6 and 7?

With plans in #30462 to test more database vendors and versions that would add further jobs to each build, though as also mentioned in that ticket we have the ability to run "daily jobs" so specific build configurations could be ran only once per day which could allow us to optimise the current build job matrix.

This is probably a meta topic. :)

I think this is fine as a #core ticket :)

Interesting reading: ​https://www.drupal.org/drupalorg/blog/drupalci-continuous-integration-testing-for-drupalorg

This was a good read, thanks for that +1

Of course, things run mostly concurrent right now, so some of those may not help at all. Especially when the biggest slow down is waiting for everything to pass.

We are kind of stretching the bounds of free Travis server power.

Yes, we are, also in #30462 is this:
Replying to nacin:

Once we switch over the GitHub repos and use the official wordpress/wordpress repo on Travis (later December, I expect), setting up a paid account is not a problem. We can also do that temporarily with a particular account now.

Travis-CI.com has a paid plan that offers "10 Concurrent jobs" ​https://travis-ci.com/plans, currently with the free Travis-CI.org we get 4 concurrent jobs. 10 jobs would cover the current build job matrix and reduce build time from the current ~30 minutes to ~15 minutes. (I'm not sure of the Travis CI hardware specs in the pro vs open source differs)

[johnbillion]

Travis has just announced native support for cancelling pending builds within branches: ​https://blog.travis-ci.com/2017-03-22-introducing-auto-cancellation

I've just enabled this on ​aaronjorbin/develop.wordpress.