WordPress.org

Make WordPress Core

Opened 4 years ago

Closed 4 years ago

#33592 closed task (blessed) (fixed)

Unicode 8.0 Emoji

Reported by: pento Owned by: pento
Milestone: 4.4.1 Priority: normal
Severity: normal Version:
Component: Emoji Keywords: emoji fixed-major
Focuses: Cc:
PR Number:

Description

Unicode 8 was finalised in June, making the new emoji from there official. We should add support for the new emoji.

Twemoji hasn't added support yet, and there's currently no indication of when they will. So, our options are:

  • Reach out to any contacts we have within Twitter to see if they can share their plans.
  • Update the Twemoji images ourselves, or contract the original artists, The Iconfactory.
  • Evaluate other Open Source emoji sets, such as Emoji One or emojidex.

We'll probably need to look at checking what browsers support the various parts of Unicode 8, too.

Attachments (6)

diverse-santa.png (92.1 KB) - added by pento 4 years ago.
diverse-santa.2.png (83.4 KB) - added by pento 4 years ago.
svg-diversity.zip (807.5 KB) - added by pento 4 years ago.
svg.sh (3.0 KB) - added by pento 4 years ago.
svg-diversity.2.zip (968.8 KB) - added by pento 4 years ago.
svg.2.sh (3.8 KB) - added by pento 4 years ago.

Download all attachments as: .zip

Change History (42)

#1 @pento
4 years ago

  • Owner set to pento
  • Status changed from new to assigned

#2 follow-up: @pento
4 years ago

@joen - given your original work on emoji, do you have opinions on these options?

#3 in reply to: ↑ 2 ; follow-up: @Joen
4 years ago

Replying to pento:

@joen - given your original work on emoji, do you have opinions on these options?

I do, and thank you for the ping!

Drawing the images ourselves is not only a lot of work for very skilled artists, but will very likely be duplicate work as I'm confident Unicode 8 emoji will eventually hit Twemoji — emoji are simply too popular for the set not to be expanded to include the cool new stuff.

Reaching out to Twitter would be great if anyone has contacts, though I'm unsure what'll come of that. Seems like the expanded set will probably drop at an unknown time when Twitter considers it a priority, and I'm unsure whether us pinging them will move the needle.

Filling in the 8.0 range with temporary replacements from Emoji One or Emojidex I'm not personally a fan of — while both sets are gorgeous on their own, they feel visually incompatible with Twemoji, though of the two Emoji One is definitely closer if we're antsy.

"Wait and see" is a lazy thing to say, I know, but at the moment that's what I'd personally do. That being said, I think improvements to the current set could be super worthwhile. There was a diversity upgrade recently, which I don't seem to remember us supporting yet — when emoji are inserted via iOS, the emoji shows up with a skin-color square next to it. Seems like getting that to work could be a nice iteration while waiting for the expanded set, maybe.

@pento
4 years ago

#4 @pento
4 years ago

I wrote an experimental modification to svg.sh to generate skintone variations of Twemoji:

skintone() {
  local f
  local key
  local tonecode
  local tone
  local content

  local tonecodes=(1f3fb 1f3fc 1f3fd 1f3fe 1f3ff)
  local tones=(ffe0c4 e2b98f dfa675 d79e84 5b0000)

  local base=(1f385 1f466);

  for f in "${base[@]}"; do
    for key in "${!tonecodes[@]}"; do
      tonecode="${tonecodes[$key]}"
      tone="${tones[$key]}"

      content=$(
        cat "svg/$f.svg" |
        sed "s/#d79e84/#$tone/g"
      )

      echo "svg/$f-$tonecode.svg"
      echo "$content" >"svg/$f-$tonecode.svg"
    done
  done
}

Given diverse-santa.png, it obviously needs tweaking, but it's a start.

Unfortunately, that doesn't generate the PNG files, which are created from the AI files. We could probably figure out how to create them as a stop-gap until the official Twemoji update comes.

My concern with this option is that I'm not aware of any Twitter contacts that anyone has - we may be waiting some time.

Last edited 4 years ago by pento (previous) (diff)

#5 @pento
4 years ago

I'm not wild about the tones I used. So I don't have to keep searching for them, here are the tones that Apple uses:

#fadcbc
#e0bb95
#bf8468
#9b643d
#594539

@pento
4 years ago

#6 @pento
4 years ago

diverse-santa.png uses Apple's skin tones, and changes the eye colour for the darkest skin tone. Still needs tweaking, but it shows that we can tweak these things on a per-tone basis.

skintone() {
  local f
  local key
  local tonecode
  local tone
  local content

  local tonecodes=(1f3fb 1f3fc 1f3fd 1f3fe 1f3ff)
  local tones=(fadcbc e0bb95 bf8468 9b643d 594539)

  local base=(1f385);

  for f in "${base[@]}"; do
    for key in "${!tonecodes[@]}"; do
      tonecode="${tonecodes[$key]}"
      tone="${tones[$key]}"

      content=$(
        cat "svg/$f.svg" |
        sed "s/#d79e84/#$tone/g"
      )

      if [ $tone == "594539" ]
          then
            content=$(
              echo "$content" |
              sed "s/#662113/#101010/g"
            )
      fi

      echo "svg/$f-$tonecode.svg"
      echo "$content" >"svg/$f-$tonecode.svg"
    done
  done
}

@pento
4 years ago

@pento
4 years ago

#7 @pento
4 years ago

svg-diversity.zip contains a first pass at converting all of the "diversifiable" emoji. It needs a lot of tweaking. :-)

svg.sh does the conversion.

#8 follow-up: @pento
4 years ago

Things that definitely need attention:

  • Skin-tone highlights (for example, in thumbsup) need to change for each skin tone.
  • In policeman, the badge is changing colour instead of the skin.
  • Blue eye colour needs to change on the darkest 2-3 tones.
  • Blond hair needs to change on the darkest 2-3 tones.
  • Hair/eyes/mouth on the lightest and darkest baby need tweaking.
  • Red lips need tweaking on the darkest 2-3 tones.
  • Scream hands and head need a lot of work, same with sad and worried head.
  • The purple in the devil faces needs changing.

Suggestions are welcome. :-)

@pento
4 years ago

@pento
4 years ago

#9 @pento
4 years ago

In [svg-diversity.2.zip] and [svg.2.sh]:

  • Add preview.html, for easy viewing
  • Add Simpsons Yellow default skin tone
  • First pass at skin tone highlights
  • Fixed policeman
  • Fixed baby

#10 in reply to: ↑ 8 ; follow-up: @Joen
4 years ago

Replying to pento:

Suggestions are welcome. :-)

It looks like you're already doing some one-off conditionals to tune individual emoji that need attention in certian skintones. It seems to me like this is a good thing, and probably something we should do for all the edgecases you mention, like thumbs up highlights and police badges. If the script knows what SVG elements to look for and how to color them, we won't risk overriding changes downstream.

#11 in reply to: ↑ 10 @pento
4 years ago

Replying to Joen:

If the script knows what SVG elements to look for and how to color them

It's not that smart. :-)

It just search and replaces #aabbcc colour values, it doesn't try to figure out what element their in. So far, there hasn't been much in the way of crossed wires.

we won't risk overriding changes downstream.

I'm not really concerned about that. The next update to twemoji, whenever it comes, will almost certainly include Unicode 8, so we don't need to be forwards compatible.

#12 in reply to: ↑ 3 @mkaz
4 years ago

Replying to Joen:

@joen, @pento - I've reached out to our contact at Twitter.
I'll update when I hear back on their plans with Unicode 8 Emoji.

This ticket was mentioned in Slack in #core by sergey. View the logs.


4 years ago

#14 @pento
4 years ago

No status update from me.

@mkaz, any update from your Twitter contacts?

#15 @wonderboymusic
4 years ago

  • Milestone changed from 4.4 to Future Release

#16 @pento
4 years ago

In 35596:

Emoji: Add some new flags, and a handful from Unicode 8.0.

Including the following that should be used as often as possible:

U+1F1E6 U+1F1FA Flag for Australia
U+1F3CF Cricket Bat and Ball
U+1F32F Burrito

See #33592.

#17 @pento
4 years ago

  • Milestone changed from Future Release to 4.4

I'm (probably) not going to get diversity emoji done for 4.4, but I need to add a test for Unicode 8.0 emoji to wp-emoji-loader.js.

#18 @pento
4 years ago

In 35606:

Emoji: Use twemoji in browsers that don't support Unicode 8 emoji.

Some less advanced browsers are yet to add support for the important advances made in Unicode 8. Let's make ensure that their users can experience emoji in their full glory.

See #33592.

#19 @pento
4 years ago

  • Milestone changed from 4.4 to Future Release

Okay, we can continue kicking this can down the road.

#20 @pento
4 years ago

  • Milestone changed from Future Release to 4.5

Twitter just landed their Unicode 8 implementation, they've confirmed plans to update the GitHub repo.

#21 @pento
4 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 36126:

Emoji: Update emoji support to Unicode 8.0, including the Diversity set.

Fixes #33592.

#22 @pento
4 years ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

Not quite finished, investigating whether the s.w.org CDN can be flushed (some of the images were updated), or if we need to move the images to a new path.

#23 @nacin
4 years ago

  • Milestone changed from 4.5 to 4.4.1

We also need to check if the browser supports diversity, a la [35606].

And then, we should backport this to 4.4.1.

#24 @jorbin
4 years ago

This feels fairly borderline to be backported into 4.4.1. I would be fine if something can happen in the next few days, but this might also be something that needs to slip to a potential 4.4.2

#25 @kraftbj
4 years ago

Quick drive-by report while still on vacation: On Trunk, at least some native emoji is being replaced. OS X on latest stable Chrome. Example "Snowman Without Snow" (U+26C4) is being rendered natively, then replaced by Twiemoji on my prod site ( https://kraft.im/2016/01/snow-day/ ).

In another post, I can't save the hot dog emoji without it being saved in the db as an img tag. I can debug these a bit more (and spin off into a new ticket if needbe) next week when I'm back in the saddle.

#26 @pento
4 years ago

In 36160:

Emoji: Add a test to the emoji loader, to see if the browser supports emoji diversity. If it doesn't, fall back to twemoji.

See #33592.

#27 @pento
4 years ago

  • Keywords fixed-major added

Chrome should just be replacing everything - they still haven't fixed rendering colour emoji when the font-weight is greater than 500.

I wasn't able to reproduce the hot dog emoji being saved as an img tag.

#28 @pento
4 years ago

In 36161:

Emoji: Add Emoji Diversity support, and fall back to twemoji if the browser doesn't support diverse emoji.

Merge of [36126] and [36160] to the 4.4 branch.

See #33592.

#29 @pento
4 years ago

Leaving this ticket open, pending feedback from Systems as to whether we can purge the CDN or not.

#30 @ocean90
4 years ago

Shouldn't wp_encode_emoji() be updated as well?

#31 @pento
4 years ago

Nope. That regex covers (rather lazily, admittedly) the entire character space of emoji, rather than just the individual characters supported by twemoji.

There's probably a good argument that we should figure out how to keep the twemoji.js regex in sync with the wp_encode_emoji() regex.

#32 @ocean90
4 years ago

Should we clarify this in docs of wp_encode_emoji()? I was confused by the "Currently, only Unicode 7 emoji are supported." line.

#33 @pento
4 years ago

Actually, I just had another look at the regex, and it's missing a bunch of edge cases. I opened #35293 to fix it.

#34 @pento
4 years ago

  • Component changed from General to Emoji

This ticket was mentioned in Slack in #core by jorbin. View the logs.


4 years ago

#36 @pento
4 years ago

  • Resolution set to fixed
  • Status changed from reopened to closed

CDN has been purged, the magic is happening live.

Note: See TracTickets for help on using tickets.