Make WordPress Core

Opened 16 months ago

Closed 3 months ago

Last modified 3 months ago

#57600 closed task (blessed) (fixed)

Twemoji: Explore alternatives

Reported by: kraftbj's profile kraftbj Owned by: desrosj's profile desrosj
Milestone: 6.5 Priority: normal
Severity: normal Version:
Component: Emoji Keywords: has-patch has-unit-tests add-to-field-guide
Focuses: Cc:

Description

With the recent changes at Twitter, there is an uncertain path forward for Twemoji, the script and images we use for Core's emoji replacement support.

With this uncertainty in the air, WordPress should consider alternatives to this script and image library.

https://github.com/twitter/twemoji/issues/570

Change History (26)

#1 @desrosj
16 months ago

Thanks for opening this, @kraftbj!

I've been thinking about this a bit, but I don't really have a great recommendation yet.

It does look like there have been a few PRs merged this month, and a few of the contributors that were previously with the company are the ones interacting/merging these PRs.

#2 @JeffPaul
16 months ago

In a brief search I found noto-emoji though it’s licensed Apache 2.0/SIL Open Font 1.1 whereas Twemoji is MIT/CC-BY 4.0. Probably worth sitting on for now, but as we get deeper into WP 6.2 it would have been otherwise nice to have updated to Unicode/Emoji 15.

#3 @kraftbj
16 months ago

I agree, it would be ideal to have Emoji 15 support in for 6.2. I don't want to rush to a decision if 1. Twitter starts releasing updates again or 2. Some other path for Twemoji to live on happens.

I don't think we should wait too long, though, so WP 6.3 should have Emoji 15 support somehow, one way or another.

If it is just an Twemoji update, I would support it in 6.2.x if the powers that be are okay with that. If it is moving to a different library, then 6.3 no doubt.

#4 @azaozz
16 months ago

In a brief search I found noto-emoji though it’s licensed Apache 2.0/SIL Open Font 1.1

Seems there are Noto Emoji (https://github.com/fontsource/fontsource/tree/main/fonts/google/noto-emoji) and Noto Color Emoji (https://github.com/fontsource/fontsource/tree/main/fonts/google/noto-color-emoji).

Afaik the fonts with SIL Open Font license can be used with GPLv2 code (they are not "merged" or part of the code, they are distributes alongside the code). Perhaps Noto Color Emoji would be a good alternative.

Also, related #52219.

Last edited 16 months ago by azaozz (previous) (diff)

#5 follow-up: @desrosj
14 months ago

It looks like the previous maintainers (though no longer employed by Twitter) have created a fork with the full intention of supporting it going forwards.

From an issue asking for clarification on the purpose of the new repo:

" The original Twemoji was maintained by 2 Twitter employees, @sofodesign and me. With Elon Musk's acquisition of Twitter, we both no longer work there, nor do we have commit permissions to the original repo anymore. There's no indication at all from Twitter's side whether they intend to keep maintaining the original Twemoji; given Sofo and I did it in essentially our free time, we imagine it's highly unlikely and probably pretty low on the priority list of anyone there right now. We (and other past Twitter employees who worked on Twemoji and left well before the Musk acquisition) don't want Twemoji to go to the big Internet graveyard in the sky, so we've forked the original Twemoji here and plan to continue working on it (releasing new Unicode-approved emoji, maintenance, etc) despite no longer being employed by Twitter. I hope that clarifies things; if not, feel free to reply with any further questions."

There's also an issue dedicated to adding support for Unicode 15. The issue does say the hope is to have something in March (which has already passed), but it's made clear that the maintainers require some support in order to dedicate more time to maintaining the project.

I think this is likely our best option going forward.

#6 @JeffPaul
14 months ago

@desrosj I've been tracking this as well and concur with your assessment, though I would propose that we wait and see that the jdecked/twemoji fork actually gets a new Unicode 15 release out at some point before working on any changes here in core. I'm subscribed and following that fork and while they have some minor releases, I would prefer to wait and see that they get to v15 support before considering a switch.

#7 @desrosj
14 months ago

@JeffPaul Sure, I wasn't clear above. But that is also my suggestion. There is nothing worth including currently, but it is a compelling potential library to adopt in the future when there are significant enough modifications.

#8 @kraftbj
14 months ago

I concur with the above. Let's keep an eye on it and if it sticks, let's use it.

We can also consider and be aware of other options out there, but all things being equal, unless something else truly brings something different to the table, let's use the original maintainer's version.

#9 in reply to: ↑ 5 @Hareesh Pillai
14 months ago

I came here to reply about this fork, and it seems @desrosj beat me to it. :)

It looks like the previous maintainers (though no longer employed by Twitter) have created a fork with the full intention of supporting it going forwards.

This fork seems like a viable alternative as the license is kept intact.

I would like to add that the new fork has a release that contains the changes (made at the old Twemoji repository) to replace MaxCDN with jsDelivr. This was done as MaxCDN was retired by the company that acquired it.

PS: Should we update this ticket's component to External Libraries?

#10 @desrosj
14 months ago

I think leaving this in the Emoji component is fine. Historically, even though Twemoji is an external library, we've grouped related tickets into the Emoji component instead.

#11 @desrosj
6 months ago

  • Milestone changed from Future Release to 6.5

Looks like the fork mentioned above has released version 15.0.2 with support for Unicode 15. 🎉

I have not yet confirmed license compatibility or whether we can continue to use npm to install. But an encouraging, unexpected development!

#12 @kraftbj
6 months ago

This is very encouraging. I'll put this on my list to test out. The license appears to be MIT and CC Attribution.

This ticket was mentioned in PR #5804 on WordPress/wordpress-develop by @desrosj.


5 months ago
#13

  • Keywords has-patch has-unit-tests added

This switches the source for the Twemoji library from twitter/twemoji to jdecked/twemoji while also applying the 15.0.3 update.

Trac ticket: https://core.trac.wordpress.org/ticket/57600

@peterwilsoncc commented on PR #5804:


4 months ago
#14

This looks good to me, the test can probably be a blackbird as it uses a ZWJ https://emojipedia.org/black-bird

#15 follow-up: @desrosj
4 months ago

@dd32 would you be able to get https://github.com/jdecked/twemoji/releases/tag/v15.0.3 committed to the .org CDN?

#16 in reply to: ↑ 15 @dd32
4 months ago

Replying to desrosj:

dd32 would you be able to get https://github.com/jdecked/twemoji/releases/tag/v15.0.3 committed to the .org CDN?

Done! Test image:
https://s.w.org/images/core/emoji/15.0.3/72x72/1f426-200d-2b1b.png

@peterwilsoncc commented on PR #5804:


4 months ago
#17

@desrosj In 63c1fa5b7eebbef0191394c0c22e1890d3bed4df I've pushed the update to the JavaScript tests.

On a system that doesn't support the new emoji, the canvas renders identically
https://github.com/WordPress/wordpress-develop/assets/519727/33fc163b-7526-4a7c-87c3-57e9214b8033

Using the emoji in a post renders on a system without support renders the emoji correctly
https://i0.wp.com/github.com/WordPress/wordpress-develop/assets/519727/5f1d0ccf-55a0-4ab9-9c4e-9ddd7625e1f7

On a system that does support the new emoji, the canvas renders differently
https://i0.wp.com/github.com/WordPress/wordpress-develop/assets/519727/745ca048-d449-4fc4-b233-14fd0e83c9f1

#18 @peterwilsoncc
4 months ago

So it's documented somewhere, this is the utility I used to get the correct codes for the new JavaScript test https://it-tools.tech/emoji-picker

@desrosj commented on PR #5804:


4 months ago
#19

Thanks @peterwilsoncc! I started to do some precommit testing on this earlier today before I helped get the update out, and I noticed that the grunt precommit:emoji script is broken. I tried several different forms of the URL, including changing it to a version specific one, but couldn't get it to work. I'll check in some more my tomorrow, but if you care to have a look during your day, feel free.

@peterwilsoncc commented on PR #5804:


4 months ago
#20

@desrosj Looks to be due to GitHub sunsetting SVN support. The grunt file is trying to use svn to get a list of the files without doing a clone.

https://github.com/WordPress/wordpress-develop/blob/cf8b74de164f322a78ea51921b236213805e6f79/Gruntfile.js#L1056

@peterwilsoncc commented on PR #5804:


4 months ago
#21

In https://github.com/WordPress/wordpress-develop/pull/5804/commits/21cfb61661e405f8a422a5172937c520dfec17a0 I've pushed a fix to the precommit script that uses the GitHub CLI instead of SVN.

It appears to be working but I haven't pushed the changes to the formatting file as it would be good to get a logic check on that from you.

@peterwilsoncc commented on PR #5804:


4 months ago
#24

Precommit broken due to the number of files exceeding 1000.

I've tried using the graphql API and it works locally but the grunt.spawn command I've written is broken but should show what I am trying to do.

#25 @desrosj
3 months ago

  • Owner set to desrosj
  • Resolution set to fixed
  • Status changed from new to closed

In 57626:

Emoji: Replace twitter/twemoji with jdecked/twemoji.

After a chaotic change of ownership, the twitter/twemoji library is now considered abandoned.

After waiting for this moment to arise, a fork was created by several former employees who had maintained the library which lives at jdecked/twemoji on GitHub.

This switches out where the underlying source code comes from for the library, and applies the 15.0.3 update, which adheres to the Unicode 15 spec and adds support for all Emoji introduced in Emoji 15.0.

This does not update the underlying precommit:emoji Grunt script responsible for updating formatting.php. After GitHub recently sunset support for SVN, the current process needs to be replaced with a new one. This will be handled in #60520.

Let the masses rejoice for the 🐦‍⬛ singing in the dead of night, secure 🛜, aromatic 🫚, and some very silly 🪿🪿.

RIP Twemoji. Long live Twemoji!

Props peterwilsoncc, kraftbj, jeffpaul, azaozz, dd32, hareesh-pillai, jorbin.
Fixes #57600.

@desrosj commented on PR #5804:


3 months ago
#26

Committed in Core-57626. I've moved all build related changes to #6107 for further exploration.

#27 @stevenlinx
3 months ago

  • Keywords add-to-field-guide added

#28 @desrosj
3 months ago

In 57758:

Build/Test Tools: Fix the precommit:emoji script.

GitHub recently sunset support for Subversion, causing the precommit:emoji Grunt script to break. Since there’s no direct replacement for svn ls in Git, this has been replaced with a query through the GitHub CLI.

This also adds a step in the workflow that tests the build process to run the precommit:emoji script to ensure no changes to built files are missed when updating the Twemoji library in the future.

Follow up to [57626].

Props kraftbj, peterwilsoncc, swissspidy.
Fixes #60520. See #57600.

Note: See TracTickets for help on using tickets.