Make emojis accessible

[afercia]

Splitting this out from #37432.

Currently, the WordPress Emojis are not accessible. The alt text contains the emoji character (vs a HTML encoded entity, or the emoji description) and that's a deliberate implementation choice, see 37433#comment:3 intended to allow copy and pasting the emojis.

We've discussed a bit this issue in the accessibility weekly meeting and agreed there's room for improvements. At least, starting a discussion about emojis accessibility would be very welcome.

Quickly checking how screen readers announce the WordPress emojis, there's no accessible name or description they can use:

Note: VoiceOver reads out "group" because the image is a svg file.

Just adding an aria-label attribute (I've quickly edited in the browser console) gives screen readers an accessible name to announce:

Comparing with what others do, for example Twitter because they usually have great solutions for accessibility, they use png images and both an aria-label and a title attribute, so VoiceOVer in this example reads out both. I guess the title attribute is used just to show the browser's "tooltip" with the emoji name:

<img class="Emoji Emoji--forText" src="https://abs.twimg.com/emoji/v2/72x72/1f60e.png" 
 draggable="false" alt="" title="Smiling face with sunglasses" aria-label="Emoji: Smiling face with sunglasses">

The aria-label solution is mentioned also on a related issue on the twemoji GitHub: ​https://github.com/twitter/twemoji/issues/41

[dd32]

Loading descriptions client-side of all emoji's is unlikely to be a solution which we can use, simply due to the sheer size of emoji's and the lengths of their translations (which also then need to be translated).

Can we make it display the Emoji itself to screen readers instead of group? (Which was also an intention I believe) We'll then be throwing emoji descriptions into the ball court of screen readers though (many of which may not support the full emoji language for quite some time)

[peterwilsoncc]

For the SVG copies (used by 97% of web traffic) is there a server side solution we could implement by adding the descriptions to the SVGs on the CDN?

@afercia how do screen readers handle native emoji?

[afercia]

Screen readers usually read out what browsers expose to them through their accessibility APIs. Sometimes, in limited circumstances, they try to read directly the DOM to work around known browsers inabilities to correctly expose accessible properties. So it's mostly a matter of platform/browsers support for unicode/svg and how browser expose these informations to assistive technologies. However, WordPress is not using inline SVGs. These are just standard <img> elements with no textual alternative whatsoever.

As per the group thing, seems specific to Safari+VoiceOver and I think just adding a role="img" attribute could help the emojis to be correctly identified as images. Quick test available on this codepen: ​http://s.codepen.io/afercia/debug/wWXQao

Native support:
interestingly, VoiceOver announces the WP Emojis skin tone modifier as "Emoji modifier fitzpatrick type" and displays it in the Caption Panel:

hearing just "Emoji modifier fitzpatrick type" is a bit annoying and not useful at all, one more reason to provide a textual alternative in my opinion. This made me curious though so I've tested a bit more. Seems Safari+VoiceOver on OS X El Capitan can correctly announce the native emoji characters. I guess no surprise, since this platform has a very good support for Unicode. By the way, it seems they're often announced with a slightly different name, for example "Station" is announced as "Subway station" and this happens in many, many, other cases:

Unfortunately, on other platforms support greatly varies across browsers even for common Unicode characters, see the test for the "right-pointing double angle quotation mark" on the codepen. In the worst case, emojis are announced as "symbol" and then an alphanumeric string (often different from the character code).

Descriptions size/Translations:
yep, I can understand the technical challenge. I really have no idea about the current implementation details :) About translations, maybe there are already some translations available? See below:

[afercia]

Update:
about supported/unsupported emoji versions, Safari+VoiceOver announce the supported one:

the first one in this example is announced as "slightly smiling face". Setting VoiceOver in a different language, for example Italian, works and the emoji is announced as "Faccina leggermente sorridente". Nope for the unsupported version 9 fallback to image though.

[pento]

I've been thinking about this a little bit, and I think I have a direction worth investigating.

• First, change the emoji element from an img to an object. This allows the SVG to interact with the embedding window. (The CDN currently doesn't allow embedding this way due to X-Frame-Options, but that can be fixed.) In the mean time, ​here's an SVG with the below script added, and no X-Frame-Origins restrictions.
• Modify all of the SVGs to add a little bit of JavaScript. Something like:

<script type="text/javascript">
    window.parent.postMessage( { description: 'Rainbow flag', emoji: '1f3f3-fe0f-200d-1f308.svg' }, '*' );
</script>

• Listen for this message in wp-emoji.js:

window.addEventListener( 'message', descriptionListener, false ):
function descriptionListener( event ) {
    // Check that the message is coming from the CDN, and is correctly formatted.

    // Find all emoji elements that match event.data.emoji

    emojiEl.setAttribute( 'title', event.data.description );
    emojiEl.setAttribute( 'aria-label', 'Emoji: ' + event.data.description );
}

This idea has several weaknesses:

• It only works for SVG embeds, not PNG embeds. That excludes IE 8 and earlier.
• To be properly translatable, we'd need to have different files for each language. It's not practical to update a single SVG file each time a new translation is added, due to the way the CDN works. Effectively, we'd have 1f3f3-fe0f-200d-1f308.en_AU.svg, 1f3f3-fe0f-200d-1f308.it_IT.svg, etc. I'm not sure whether we'd load emoji by site language or user client language, but that's an easy argument to have. :-)
• It updates the DOM twice - once to add the emoji image, once to add the aria-label. I don't know how this affects screen readers.
• Adding the title attribute doesn't cause a hover tooltip on object elements. I don't know how the aria-label behaves.
• I don't know if this would be accepted by the upstream Twemoji project. It's unlikely that we'd need to change twemoji.js, so the primary concern is additional maintenance when emoji need to be updated - we'd need a script to add the <script> to all of them. Also, Twitter may not share their translations, so we may need to add all of the strings to GlotPress.

Finally, here are some things that don't work, while I was experimenting:

• Embedding as an img. The SVG can't send the message.
• Accessing the SVG DOM from the parent window, so that we don't need to do the postMessage. Cross Domain Origin restrictions.

[pento]

I don't think there's any sort of realistic method for us to maintain translations for all emoji descriptions. This would add ~20k words to the translation files, in addition to causing all sorts of issues trying to load them in a fast and bandwidth-efficient manner.

37486.diff​ adds an aria-label and role="img" to each emoji <img> element, which changes the behaviour slightly in Safari/VoiceOver, but otherwise doesn't seem like much of an improvement.

Unless there are any better ideas, I'm inclined to close this issue, as it's not something we can make progress on.

[afercia]

Thanks for the patch!

Testing on macOS Mojave 10.14.5
Safari 12.1.1
VoiceOver 9

First thing to note, Safari and VoiceOver now announce the "carrot" emoji. Three years ago they didn't. Progress! :) This seems to confirm that with operating systems / browsers updates, new emoji versions are natively supported and recognized.

With the patch applied, VoiceOver announces "image" instead of "group", which is a welcomed improvement. The aria-label doesn't appear to make a difference. More on this later.

Worth noting the actual behavior varies depending on the command used to read content, for example:

Read All: Control + Option + A
reads the emojis, treating them as images

Read Paragraph: Control + Option + P
ignores the emojis and reads just the text: "Hello, this is a and this is a". Note: this is the standard behavior also with inline images.

Read Sentence: Control + Option + S
Same.

Read Line: Control + Option + L
Same.

Navigating content with Control + Option + Right (and Left) arrows:
reads the emojis, treating them as images

Testing on Windows 10 Pro – version 1803 build 17134.829
Chrome 75.0.3770.100
Firefox 67.0.4
Internet Explorer 11
JAWS 2018
NVDA 2019.1.1

Before the patch:

• Firefox JAWS: "Hello, this is a and this is a"
• Chrome JAWS: "Hello, this is a and this is a"
• IE 11 JAWS: "Hello, this is a graphic slightly smiling face and this is a graphic" note: doesn't read automatically the whole paragraph though and requires to use arrows to navigate through content

So it appears JAWS 2018 doesn't announce the emojis with modern browsers and surprisingly announce at least the smile with an old browser like IE 11, probably because it fallbacks to scraping the DOM for IE 11?

• Firefox NVDA: "Hello, this is graphic slightly smiling face and this is a graphic carrot"
• Chrome NVDA: "Hello, this is graphic slightly smiling face and this is a graphic carrot"
• IE11 NVDA: "Hello, this is graphic slightly smiling face and this is a graphic carrot"

NVDA works very well with all the tested browsers.

With the patch applied:

• JAWS with all the 3 browsers: no difference
• NVDA with all the 3 browsers: no difference

Looking at the Chrome dev tools > Accessibility tab helps to understand why the aria-label makes no difference:

Before the patch, the accessible name of the emoji come from the alt attribute (which by the way contains the emoji so it relies on the fact the emoji is recognized by the OS and browser):

After the patch, the accessible name comes from the aria-label which just overrides the alt attribute:

So it appears the aria-label is redundant and can be removed.

Technically, also the ARIA role is redundant because <img> element have a native role of... image. However, it helps VoiceOver to correctly announce "image" instead of "group" and I'd lean towards keeping it. HTML validators may warn this role is redundant but it doesn't harm anything.

Lastly, I do realize there are no realistic methods for further improvements, so far. However, I'd like to keep this ticket open to track the issue and future progress in OSes support. If no objections, I'd like to propose to make this ticket a blessed task.

[afercia]

Update: I've just upgraded to latest Windows (Windows 10 Pro – version 1903 build 18362.207) and JAWS 2019. Things work far better and JAWS correctly reads the emojis on all browsers.

See the JAWS speech history in the screenshot below. Note: it displays the emojis as images but they're actually announced as "slightly smiling face" and "carrot":

while with JAWS 2018 they were completely ignored:

[pento]

Thank you for the extensive testing, @afercia! 🙂

37486.2.diff​ removes the redundant aria-label.

I'm fine with changing this ticket to a task, if you'd like to use it for ongoing tracking of OS/browser/screen reader behaviour.

[afercia]

In 45726:

Accessibility: Improve the way emojis are announced by screen readers.

Adds an ARIA role="img" to the emojis markup in the front end. While the ARIA role is technically redundant, it helps Safari and VoiceOver to properly announce the emojis as "image". Without the role, they announce a generic "group" role.
Tested with Safari 12.1.1 / VoiceOver 9.

Props pento.
See #37486.

[desrosj]

Re-adding the milestone to this ticket.

@afercia Are there any new developments here that would require further changes? If not, can this be closed out in favor of new individual tickets for additional problems?

[afercia]

@desrosj the fundamental problem is in the way the various operating system expose emojis to assistive technologies. Some sets of emojis are exposed correctly so that their name is available to assistive tech. When new sets of emojis get added, this is often a problem though, as they're not exposed correctly. They would need a translatable alt text, which is technical challenging.

It was agreed to keep this ticket open as blessed task for ongoing tracking of OS/browser/screen reader behaviour so I'd like to keep it open. Ça va sans dire that ongoing tracking would require time, which is... very limited :)

• relevent to a 4 year old issue in trac
• strictly speaking, this type of functionality should really be added into the upstream twemoji library
  • in ​`function parseNode(node, options)`

regarding the actual code:

• I mixed some ES6 into the file
  • I wasn't sure if it gets transpiled with Babel during the build process
  • I could re-write it to only use ES5
    • please let me know if that's necessary