WordPress.org
Get WordPress

Make WordPress Core

Changeset 59508


Ignore:
Timestamp:
12/13/2024 03:32:55 PM (5 weeks ago)
Author:
desrosj
Message:

Build/Test Tools: Trim down the upgrade testing matrix.

The upgrade testing workflow is currently at ~978 jobs spawned from the strategy matrix. While it's great to test all possible combinations, GitHub's UI cannot keep up with tracking that number of jobs, often taking 30-45 minutes to accurately report the outcome even though the jobs themselves all complete in under 5 minutes.

This is 2x the total number of concurrent jobs allowed for the entire organization (which creates a backlog and slows everything down even more).

This trims down the number of combinations included in the testing matrices to be a bit more thoughtful following the following methodology:

  • The last two releases of WordPress are tested against all PHP and MySQL LTS version combinations and the most recent innovation release.
  • The next 6 oldest versions of WordPress are tested against both the oldest and newest releases of PHP currently supported for both PHP 7 & 8 along with the oldest and newest MySQL LTS versions currently supported (no innovation releases).
  • For the remaining versions of WordPress receiving security updates, they are only included if the database version was different that the previous major release.
  • The oldest version of WordPress receiving security updates should always be tested against the same full list of combinations as the last two releases.

When choosing which MySQL versions to test against:

  • Only the most recent innovation release should be included in testing.
  • Even though MySQL >= 5.5.5 is currently supported, there are no 5.5.x Docker containers available that work on modern architectures.
  • 5.6.x Docker containers are available and work, but 5.6 only accounts for ~2.3% of installs as of 12/6/2024.defaults:
  • 5.7.x accounts for ~20% of installs, so this is used below instead.

With these changes, the total number of jobs is reduced by ~58%.

Props johnbillion, mukesh27.
See #62221.

File:
1 edited

Legend:

Unmodified
Added
Removed

  • trunk/.github/workflows/upgrade-testing.yml

    r59403 r59508  
     1# Confirms that updating WordPress using WP-CLI works successfully.
     2#
     3# This workflow is not meant to test wordpress-develop checkouts, but rather tagged versions officially available on WordPress.org.
    14name: Upgrade Tests
    25
    … …  
    3235permissions: {}
    3336
     37# Because the number of jobs spawned can quickly balloon out of control, the following methodology is applied when
     38# building out the matrix below:
     39#
     40# - The last two releases of WordPress are tested against all PHP/MySQL LTS version combinations and the most recent
     41#   innovation release.
     42# - The next 6 oldest versions of WordPress are tested against both the oldest and newest releases of PHP currently
     43#   supported for both PHP 7 & 8 along with the oldest and newest MySQL LTS versions currently supported (no innovation
     44#   releases). At the current 3 releases per year pace, this accounts for 2 additional years worth of releases.
     45# - Of the remaining versions of WordPress still receiving security updates, only test the ones where the database
     46#   version was updated since the previous major release.
     47# - The oldest version of WordPress receiving security updates should always be tested against the same combinations as
     48#   detailed for the last two releases.
     49
     50#  Notes about chosen MySQL versions:
     51#    - Only the most recent innovation release should be included in testing.
     52#    - Even though MySQL >= 5.5.5 is currently supported, there are no 5.5.x Docker containers available that work on
     53#      modern architectures.
     54#    - 5.6.x Docker containers are available and work, but 5.6 only accounts for ~2.3% of installs as of 12/6/2024.defaults:
     55#    - 5.7.x accounts for ~20% of installs, so this is used below instead.
    3456jobs:
    35   # Spawns upgrade testing from WordPress 6.x versions on PHP 8.x with MySQL.
    36   upgrade-tests-wp-6x-php-8x-mysql:
     57  # Tests the full list of PHP/MySQL combinations for the last two versions of WordPress.
     58  upgrade-tests-last-two-releases:
    3759    name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
    3860    uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
    … …  
    4466      matrix:
    4567        os: [ 'ubuntu-latest' ]
    46         php: [ '8.0', '8.1', '8.2', '8.3', '8.4' ]
    47         db-type: [ 'mysql' ]
    48         db-version: [ '5.7', '8.0', '8.4' ]
    49         wp: [ '6.0', '6.1', '6.2', '6.3', '6.4', '6.5', '6.6', '6.7' ]
    50         multisite: [ false, true ]
    51 
    52     with:
    53       os: ${{ matrix.os }}
    54       php: ${{ matrix.php }}
    55       db-type: ${{ matrix.db-type }}
    56       db-version: ${{ matrix.db-version }}
    57       wp: ${{ matrix.wp }}
    58       new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
    59       multisite: ${{ matrix.multisite }}
    60 
    61   # Spawns upgrade testing from WordPress 6.x versions on PHP 7.x with MySQL.
    62   upgrade-tests-wp-6x-php-7x-mysql:
     68        php: [ '7.2', '7.3', '7.4', '8.0', '8.1', '8.2', '8.3', '8.4' ]
     69        db-type: [ 'mysql' ]
     70        db-version: [ '5.7', '8.0', '8.4', '9.1' ]
     71        wp: [ '6.6', '6.7' ]
     72        multisite: [ false, true ]
     73
     74        exclude:
     75          - php: '7.2'
     76            db-version: '8.4'
     77          - php: '7.3'
     78            db-version: '8.4'
     79          # MySQL 9.0+ will not work on PHP 7.2 & 7.3
     80          - php: '7.2'
     81            db-version: '9.1'
     82          - php: '7.3'
     83            db-version: '9.1'
     84    with:
     85      os: ${{ matrix.os }}
     86      php: ${{ matrix.php }}
     87      db-type: ${{ matrix.db-type }}
     88      db-version: ${{ matrix.db-version }}
     89      wp: ${{ matrix.wp }}
     90      new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
     91      multisite: ${{ matrix.multisite }}
     92
     93  # Tests the remaining 6.x releases on the oldest and newest supported versions of PHP 7 & 8.
     94  upgrade-tests-wp-6x-mysql:
    6395    name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
    6496    uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
    … …  
    70102      matrix:
    71103        os: [ 'ubuntu-latest' ]
    72         php: [ '7.2', '7.3', '7.4' ]
    73         db-type: [ 'mysql' ]
    74         db-version: [ '5.7', '8.0', '8.4' ]
    75         wp: [ '6.0', '6.1', '6.2', '6.3', '6.4', '6.5', '6.6', '6.7' ]
    76         multisite: [ false, true ]
    77 
    78         exclude:
    79           - php: '7.2'
    80             db-version: '8.4'
    81           - php: '7.3'
    82             db-version: '8.4'
    83     with:
    84       os: ${{ matrix.os }}
    85       php: ${{ matrix.php }}
    86       db-type: ${{ matrix.db-type }}
    87       db-version: ${{ matrix.db-version }}
    88       wp: ${{ matrix.wp }}
    89       new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
    90       multisite: ${{ matrix.multisite }}
    91 
    92   # Spawns upgrade testing from WordPress 5.x versions on PHP 7.x with MySQL.
     104        php: [ '7.2', '7.4', '8.0', '8.4' ]
     105        db-type: [ 'mysql' ]
     106        db-version: [ '5.7', '8.4' ]
     107        wp: [ '6.0', '6.1', '6.2', '6.3', '6.4', '6.5' ]
     108        multisite: [ false, true ]
     109
     110        exclude:
     111          - php: '7.2'
     112            db-version: '8.4'
     113          - php: '7.3'
     114            db-version: '8.4'
     115    with:
     116      os: ${{ matrix.os }}
     117      php: ${{ matrix.php }}
     118      db-type: ${{ matrix.db-type }}
     119      db-version: ${{ matrix.db-version }}
     120      wp: ${{ matrix.wp }}
     121      new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
     122      multisite: ${{ matrix.multisite }}
     123
     124  # Tests 5.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 7.
    93125  upgrade-tests-wp-5x-php-7x-mysql:
    94126    name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
    … …  
    99131      matrix:
    100132        os: [ 'ubuntu-latest' ]
    101         php: [ '7.2', '7.3', '7.4' ]
    102         db-type: [ 'mysql' ]
    103         db-version: [ '5.7', '8.0', '8.4' ]
    104         wp: [ '5.0', '5.1', '5.2', '5.3', '5.4', '5.5', '5.6', '5.7', '5.8', '5.9' ]
    105         multisite: [ false, true ]
    106 
    107         exclude:
    108           - php: '7.2'
    109             db-version: '8.4'
    110           - php: '7.3'
    111             db-version: '8.4'
    112     with:
    113       os: ${{ matrix.os }}
    114       php: ${{ matrix.php }}
    115       db-type: ${{ matrix.db-type }}
    116       db-version: ${{ matrix.db-version }}
    117       wp: ${{ matrix.wp }}
    118       new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
    119       multisite: ${{ matrix.multisite }}
    120 
    121   # Spawns upgrade testing from WordPress 5.x versions on PHP 8.x with MySQL.
     133        php: [ '7.2', '7.4' ]
     134        db-type: [ 'mysql' ]
     135        db-version: [ '5.7', '8.4' ]
     136        wp: [ '5.0', '5.1', '5.3', '5.4', '5.5', '5.6', '5.9' ]
     137        multisite: [ false, true ]
     138
     139        exclude:
     140          - php: '7.2'
     141            db-version: '8.4'
     142          - php: '7.3'
     143            db-version: '8.4'
     144    with:
     145      os: ${{ matrix.os }}
     146      php: ${{ matrix.php }}
     147      db-type: ${{ matrix.db-type }}
     148      db-version: ${{ matrix.db-version }}
     149      wp: ${{ matrix.wp }}
     150      new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
     151      multisite: ${{ matrix.multisite }}
     152
     153  # Tests 5.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 8.
    122154  #
    123155  # WordPress 5.0-5.2 are excluded from PHP 8+ testing because of the following fatal errors:
    … …  
    132164      matrix:
    133165        os: [ 'ubuntu-latest' ]
    134         php: [ '8.0', '8.1', '8.2', '8.3', '8.4' ]
    135         db-type: [ 'mysql' ]
    136         db-version: [ '5.7', '8.0', '8.4' ]
    137         wp: [ '5.3', '5.4', '5.5', '5.6', '5.7', '5.8', '5.9' ]
    138         multisite: [ false, true ]
    139     with:
    140       os: ${{ matrix.os }}
    141       php: ${{ matrix.php }}
    142       db-type: ${{ matrix.db-type }}
    143       db-version: ${{ matrix.db-version }}
    144       wp: ${{ matrix.wp }}
    145       new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
    146       multisite: ${{ matrix.multisite }}
    147 
    148   # Spawns upgrade testing from WordPress 4.x versions on PHP 7.x with MySQL.
     166        php: [ '8.0', '8.4' ]
     167        db-type: [ 'mysql' ]
     168        db-version: [ '5.7', '8.4' ]
     169        wp: [ '5.3', '5.4', '5.5', '5.6', '5.9' ]
     170        multisite: [ false, true ]
     171    with:
     172      os: ${{ matrix.os }}
     173      php: ${{ matrix.php }}
     174      db-type: ${{ matrix.db-type }}
     175      db-version: ${{ matrix.db-version }}
     176      wp: ${{ matrix.wp }}
     177      new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
     178      multisite: ${{ matrix.multisite }}
     179
     180  # Tests 4.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 7.
     181  #
     182  # The oldest version of WordPress receiving security updates should always be tested.
    149183  upgrade-tests-wp-4x-php-7x-mysql:
    150184    name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
    … …  
    155189      matrix:
    156190        os: [ 'ubuntu-latest' ]
    157         php: [ '7.2', '7.3', '7.4' ]
    158         db-type: [ 'mysql' ]
    159         db-version: [ '5.7', '8.0', '8.4' ]
    160         wp: [ '4.1', '4.2', '4.3', '4.4', '4.5', '4.6', '4.7', '4.8', '4.9' ]
    161         multisite: [ false, true ]
    162 
    163         exclude:
    164           - php: '7.2'
    165             db-version: '8.4'
    166           - php: '7.3'
    167             db-version: '8.4'
    168     with:
    169       os: ${{ matrix.os }}
    170       php: ${{ matrix.php }}
    171       db-type: ${{ matrix.db-type }}
    172       db-version: ${{ matrix.db-version }}
    173       wp: ${{ matrix.wp }}
    174       new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
    175       multisite: ${{ matrix.multisite }}
    176 
    177   # Spawns upgrade testing from WordPress 4.x versions on PHP 8.x with MySQL.
     191        php: [ '7.2', '7.4' ]
     192        db-type: [ 'mysql' ]
     193        db-version: [ '5.7', '8.4' ]
     194        wp: [ '4.1', '4.2', '4.3', '4.4', '4.5', '4.6', '4.7' ]
     195        multisite: [ false, true ]
     196
     197        exclude:
     198          - php: '7.2'
     199            db-version: '8.4'
     200          - php: '7.3'
     201            db-version: '8.4'
     202    with:
     203      os: ${{ matrix.os }}
     204      php: ${{ matrix.php }}
     205      db-type: ${{ matrix.db-type }}
     206      db-version: ${{ matrix.db-version }}
     207      wp: ${{ matrix.wp }}
     208      new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
     209      multisite: ${{ matrix.multisite }}
     210
     211  # Tests 4.x releases where the WordPress database version changed on the oldest and newest supported versions of PHP 8.
     212  #
     213  # The oldest version of WordPress receiving security updates should always be tested.
    178214  #
    179215  # WordPress 4.6-4.9 are excluded from PHP 8+ testing because of the following fatal errors:
    … …  
    188224      matrix:
    189225        os: [ 'ubuntu-latest' ]
    190         php: [ '8.0', '8.1', '8.2', '8.3', '8.4' ]
    191         db-type: [ 'mysql' ]
    192         db-version: [ '5.7', '8.0', '8.4' ]
     226        php: [ '8.0', '8.4' ]
     227        db-type: [ 'mysql' ]
     228        db-version: [ '5.7', '8.4' ]
    193229        wp: [ '4.1', '4.2', '4.3', '4.4', '4.5' ]
    194230        multisite: [ false, true ]
     231    with:
     232      os: ${{ matrix.os }}
     233      php: ${{ matrix.php }}
     234      db-type: ${{ matrix.db-type }}
     235      db-version: ${{ matrix.db-version }}
     236      wp: ${{ matrix.wp }}
     237      new-version: ${{ inputs.new-version && inputs.new-version || 'latest' }}
     238      multisite: ${{ matrix.multisite }}
     239
     240  # The oldest version of WordPress receiving security updates should always be tested against
     241  # the full list of PHP/MySQL combinations.
     242  upgrade-tests-oldest-wp-mysql:
     243    name: ${{ matrix.wp }} to ${{ inputs.new-version && inputs.new-version || 'latest' }}
     244    uses: WordPress/wordpress-develop/.github/workflows/reusable-upgrade-testing.yml@trunk
     245    if: ${{ github.repository == 'WordPress/wordpress-develop' || github.event_name == 'pull_request' }}
     246    strategy:
     247      fail-fast: false
     248      matrix:
     249        os: [ 'ubuntu-latest' ]
     250        php: [ '7.2', '7.3', '7.4', '8.0', '8.1', '8.2', '8.3', '8.4' ]
     251        db-type: [ 'mysql' ]
     252        db-version: [ '5.7', '8.0', '8.4', '9.1' ]
     253        wp: [ '4.1' ]
     254        multisite: [ false, true ]
     255
     256        exclude:
     257          - php: '7.2'
     258            db-version: '8.4'
     259          - php: '7.3'
     260            db-version: '8.4'
     261          # MySQL 9.0+ will not work on PHP 7.2 & 7.3
     262          - php: '7.2'
     263            db-version: '9.1'
     264          - php: '7.3'
     265            db-version: '9.1'
    195266    with:
    196267      os: ${{ matrix.os }}
    … …  
    208279      actions: read
    209280      contents: read
    210     needs: [ upgrade-tests-wp-6x-php-8x-mysql, upgrade-tests-wp-6x-php-7x-mysql, upgrade-tests-wp-5x-php-7x-mysql, upgrade-tests-wp-5x-php-8x-mysql, upgrade-tests-wp-4x-php-7x-mysql, upgrade-tests-wp-4x-php-8x-mysql ]
     281    needs: [ upgrade-tests-last-two-releases, upgrade-tests-wp-6x-mysql, upgrade-tests-wp-5x-php-7x-mysql, upgrade-tests-wp-5x-php-8x-mysql, upgrade-tests-wp-4x-php-7x-mysql, upgrade-tests-wp-4x-php-8x-mysql, upgrade-tests-oldest-wp-mysql ]
    211282    if: ${{ github.repository == 'WordPress/wordpress-develop' && github.event_name != 'pull_request' && always() }}
    212283    with:
Note: See TracChangeset for help on using the changeset viewer.