Make WordPress Core

Opened 5 weeks ago

Last modified 8 days ago

#66015 assigned defect (bug)

Introduce Axe-core accessibility automated testing for the admin pages

Reported by: afercia Owned by: afercia
Priority: normal Milestone: 7.2
Component: Build/Test Tools Version:
Severity: normal Keywords: has-patch has-unit-tests axe-core
Cc: Focuses: accessibility, javascript, coding-standards

Description

Now that E2E tests in Core are in place, powered by PlayWright, I'd like to propose to introduce automated accessibility testing of the admin pages.

The implementation is not complex and would require:

  • Add the @axe-core/playwright package.
  • Add an E2E test that runs the axe-core checks on a set of pages.

By default, axe-core will render the page in a browser and analyze the rendered DOM.

If any interaction with a specific page is necessary to test the page in a different state, E2E test have all the tools to do so.

To be determined:

  • The set of pages to be checked.
  • The actual architectural implementation.

Change History (15)

This ticket was mentioned in ​PR #13348 on ​WordPress/wordpress-develop by ​@afercia.


5 weeks ago
#1

  • Keywords has-patch has-unit-tests added

Trac ticket: https://core.trac.wordpress.org/ticket/66015

Proof Of Concept (POF) to add axe-core accessibility tests for the admin pages.

For now, this POF only scans one page.
Re-introduces the bug fixed in https://core.trac.wordpress.org/changeset/62851 for testing purposes.

  • Make sure your local environment is correctly set up to run the E2E tests.
  • Run npm install and then build.
  • Run: npm run test:e2e -- tests/e2e/specs/accessibility.spec.js
  • Observe the test reports 1 violation (the oen intentionally re-introduced as mentioned above).
  • Observe the output:
--- Violation #1 ---
Rule ID:   select-name
Impact:    CRITICAL
Failure:   Ensure select element has an accessible name
Help:      Select element must have an accessible name
Help link: https://dequeuniversity.com/rules/axe/4.13/select-name?application=playwright
Failing elements:
  - Target Selector: #media-attachment-date-filters
  - HTML Snippet:
<select id="media-attachment-date-filters" class="attachment-filters"><option value="all">All dates</option></select>

## Use of AI Tools

None

This ticket was mentioned in ​Slack in #accessibility by joedolson. ​View the logs.


5 weeks ago

#3 @joedolson
5 weeks ago

  • Milestone Awaiting Review → 7.2

​@afercia commented on ​PR #13348:


5 weeks ago
#4

At the moment, the tests fail because there are a few accessibility violations that need to be fixed. This is exactly what we want to achieve: automated accessibility testing to catch regressions or prevent new accessibility issues from being introduced.

#5 @joedolson
4 weeks ago

  • Keywords axe-core added

​@afercia commented on ​PR #13348:


4 weeks ago
#6

It is worth noting that in the Pages sceen, the em dashes — used to visually communicate the pages hierarchy are considered by Axe-core as 'surrounding text. As such, the page links are reported as violation of the link-in-text-block rule: Ensure links are distinguished from surrounding text in a way that does not rely on color`.

https://github.com/user-attachments/assets/454892db-30ab-484a-a814-3715d044e486

We can try to ignore this violation for now.

#7 @afercia
4 weeks ago

  • Owner set to afercia
  • Status new → assigned

Note that there are three pending issues (with PRs) in the Gutenberg repo that need to be fixed and included in Core for the Axe-core checks to pass:

​https://github.com/WordPress/gutenberg/issues/82329 (merged)
​https://github.com/WordPress/gutenberg/issues/82659
​https://github.com/WordPress/gutenberg/issues/82826

Last edited 3 weeks ago by afercia (previous) (diff)

​@afercia commented on ​PR #13348:


4 weeks ago
#8

Note: once ​https://github.com/WordPress/wordpress-develop/pull/13368 is fixed, a new exception should be added to the ignores:

for the rule 'aria-allowed-role'
we should add a pattern selector '.attachments-browser .attachments-wrapper #__attachments-view-* li', to be excluded. It's another occurrence of the known false positive in axe-core, which reports as violation ARIA roles applied to <li> elements.

This ticket was mentioned in ​Slack in #accessibility by afercia. ​View the logs.


3 weeks ago

This ticket was mentioned in ​Slack in #accessibility by joedolson. ​View the logs.


3 weeks ago

​@afercia commented on ​PR #13348:


2 weeks ago
#11

Note: pages that render React components may trigger false negatives / positives simply because when the Axe-core checks run the page doesn't fully render its content yet.

For example, without the waitInterval in the configuration, the Settings - Connectors page check reports one violation:

Found 1 accessibility violation(s):

--- Violation #1 ---
Rule ID:   page-has-heading-one
Impact:    MODERATE
Failure:   Ensure that the page, or at least one of its frames contains a level-one heading
Help:      Page should contain a level-one heading
Help link: https://dequeuniversity.com/rules/axe/4.13/page-has-heading-one?application=playwright
Failing elements:
  - Target Selector: html
  - HTML Snippet:
<html class="wp-toolbar" lang="en-US">

This is a false positive. The main H1 on the page hasn't rendered yet.
Vice-versa, there are two false negatives as in: with the waitInterval in place, two legitimate violations are reported.

Will try to find a better way to detecxt when the page is fully rendered other than a magic timeout number.

This ticket was mentioned in ​Slack in #accessibility by joedolson. ​View the logs.


2 weeks ago

This ticket was mentioned in ​PR #13682 on ​WordPress/wordpress-develop by ​@afercia.


2 weeks ago
#13

See https://core.trac.wordpress.org/ticket/66015

As promised to @joedolson , here's a quick PR to test the Axe-core checks on the Revisions page.
This PR isn't meant ot be merged. It is just aimed to validate an integrate accessibility issues findings on the Revisions page.

What it does:

  • Creates a post with 3 revisions by adding 3 blocks and saving the post as draft each time.
  • Accesses the Revisions and scans it for violations.
  • Makes the page display the previous revision.
  • Scans the page again.

I will post the reported violations in a following comment.

To run the test:

  • Run npm install.
  • Run npm run test:e2e -- tests/e2e/specs/accessibility.spec.js to get the violations report in your terminal.
  • Or append the --ui flag to run the Playwright GUI:
    • npm run test:e2e -- tests/e2e/specs/accessibility.spec.js --ui
    • Once the GUI starts, click the Play button highlighted in the screenshot below.
    • Once the scan is complete, you can hover each step of the 2 tests to display the state of the UI at that time.

https://github.com/user-attachments/assets/e50d4131-8ee0-4e12-9428-29f16dcfa075

Use of AI Tools

​@afercia commented on ​PR #13682:


2 weeks ago
#14

Reported violations:

First test: 1 violation:

--- Violation #1 ---
Rule ID:   nested-interactive
Impact:    SERIOUS
Failure:   Ensure interactive controls are not nested as they are not always announced by screen readers or can cause focus problems for assistive technologies
Help:      Interactive controls must not be nested
Help link: https://dequeuniversity.com/rules/axe/4.13/nested-interactive?application=playwright
Failing elements:
  - Target Selector: html > body > #wpwrap > #wpcontent > #wpbody > #wpbody-content > .block-editor > #editor > ._0ebd6d9bf8b95794__wrapper > div > .components-snackbar-list > div > .components-snackbar-list__notice-container > .components-snackbar
  - HTML Snippet:
<div class="components-snackbar" tabindex="0" role="button" aria-label="Dismiss this notice" data-testid="snackbar">



Second test: 4 violations:

--- Violation #1 ---
Rule ID:   aria-allowed-role
Impact:    MINOR
Failure:   Ensure role attribute has an appropriate value for the element
Help:      ARIA role should be appropriate for the element
Help link: https://dequeuniversity.com/rules/axe/4.13/aria-allowed-role?application=playwright
Failing elements:
  - Target Selector: html > body > #wpwrap > #wpcontent > #wpbody > #wpbody-content > .block-editor > #editor > ._0ebd6d9bf8b95794__wrapper > div > .editor-editor-interface > .interface-interface-skeleton__editor > .interface-interface-skeleton__body > .interface-interface-skeleton__content > .editor-revisions-canvas__content > .editor-visual-editor > .components-resizable-box__container > div:nth-child(1) > .block-editor-iframe__container > .block-editor-iframe__scale-container > iframe, html > body > .editor-visual-editor__post-title-wrapper > h1
  - HTML Snippet:
<h1 contenteditable="false" class="wp-block wp-block-post-title block-editor-block-list__block editor-post-title editor-post-title__input rich-text" aria-label="Add title" role="textbox" aria-multiline="true" style="white-space: pre-wrap;">


--- Violation #2 ---
Rule ID:   aria-input-field-name
Impact:    SERIOUS
Failure:   Ensure every ARIA input field has an accessible name
Help:      ARIA input fields must have an accessible name
Help link: https://dequeuniversity.com/rules/axe/4.13/aria-input-field-name?application=playwright
Failing elements:
  - Target Selector: html > body > #wpwrap > #wpcontent > #wpbody > #wpbody-content > .block-editor > #editor > ._0ebd6d9bf8b95794__wrapper > div > .editor-editor-interface > .interface-interface-skeleton__editor > .interface-interface-skeleton__body > .interface-interface-skeleton__sidebar > .interface-complementary-area__fill > #edit-post\:document > .editor-sidebar__panel > #base-ui-\:r1h\: > .editor-post-revisions-timeline > .dataviews-picker-wrapper > .dataviews-layout__container > .dataviews-view-picker-activity
  - HTML Snippet:
<div role="listbox" aria-multiselectable="false" aria-busy="false" class="dataviews-view-picker-activity" tabindex="0" aria-activedescendant=":r8u:">


--- Violation #3 ---
Rule ID:   color-contrast
Impact:    SERIOUS
Failure:   Ensure the contrast between foreground and background colors meets WCAG 2 AA minimum contrast ratio thresholds
Help:      Elements must meet minimum color contrast ratio thresholds
Help link: https://dequeuniversity.com/rules/axe/4.13/color-contrast?application=playwright
Failing elements:
  - Target Selector: html > body > #wpwrap > #wpcontent > #wpbody > #wpbody-content > .block-editor > #editor > ._0ebd6d9bf8b95794__wrapper > div > .editor-editor-interface > .interface-interface-skeleton__editor > .interface-interface-skeleton__body > .interface-interface-skeleton__sidebar > .interface-complementary-area__fill > #edit-post\:document > .editor-sidebar__panel > #base-ui-\:r1h\: > .editor-post-revisions-timeline > .dataviews-picker-wrapper > .dataviews-layout__container > .dataviews-view-picker-activity > #\:r92\: > ._19ce0419607e1896__stack > .dataviews-view-picker-activity__item-content._19ce0419607e1896__stack > .dataviews-view-picker-activity__item-description > .editor-post-content-information > .components-truncate.css-tf28v4-Text-base-textStyles.components-text
  - HTML Snippet:
<span data-wp-c16t="true" data-wp-component="Text" class="components-truncate css-tf28v4-Text-base-textStyles components-text">2 words, 1 minute read time.</span>
  - Target Selector: html > body > #wpwrap > #wpcontent > #wpbody > #wpbody-content > .block-editor > #editor > ._0ebd6d9bf8b95794__wrapper > div > .editor-editor-interface > .interface-interface-skeleton__editor > .interface-interface-skeleton__body > .interface-interface-skeleton__sidebar > .interface-complementary-area__fill > #edit-post\:document > .editor-sidebar__panel > #base-ui-\:r1h\: > .editor-post-revisions-timeline > .dataviews-picker-wrapper > .dataviews-layout__container > .dataviews-view-picker-activity > #\:r92\: > ._19ce0419607e1896__stack > .dataviews-view-picker-activity__item-content._19ce0419607e1896__stack > .dataviews-view-picker-activity__item-fields > .dataviews-view-picker-activity__item-field > .dataviews-view-picker-activity__item-field-value > ._3896336788892095__items-row.cc39999a5fb580d4__expanded-row[data-wp-component="HStack"] > .fields-controls__author-name
  - HTML Snippet:
<span class="fields-controls__author-name">admin</span>


--- Violation #4 ---
Rule ID:   nested-interactive
Impact:    SERIOUS
Failure:   Ensure interactive controls are not nested as they are not always announced by screen readers or can cause focus problems for assistive technologies
Help:      Interactive controls must not be nested
Help link: https://dequeuniversity.com/rules/axe/4.13/nested-interactive?application=playwright
Failing elements:
  - Target Selector: html > body > #wpwrap > #wpcontent > #wpbody > #wpbody-content > .block-editor > #editor > ._0ebd6d9bf8b95794__wrapper > div > .components-snackbar-list > div > .components-snackbar-list__notice-container > .components-snackbar
  - HTML Snippet:
<div class="components-snackbar" tabindex="0" role="button" aria-label="Dismiss this notice" data-testid="snackbar">

#15 @afercia
8 days ago

  • Component Administration → Build/Test Tools
Note: See TracTickets for help on using tickets.