Zero-width space stripped by TinyMCE again

[nathanwells]

Starting from Wordpress 4.3.x TinyMCE started stripping out zero-width spaces (\u200B) from posts which essentially destroys line-break formatting in the Khmer language (especially on mobile devices). This bug was removed once already four years ago (#21025) and so it is a regression. We've alerted the TinyMCE team ​on GitHub but also wanted to alert the Wordpress team so we can see this fixed.

[iseulde]

Thanks for the report and welcome to Trac! Is this problem new to 4.6?

I see the GitHub issue was acknowledged by TinyMCE in April.

We are also using that character for the text patterns, but it is carefully inserted and removed again. We are not removing just any character. We should maybe use \uFEFF though if the other is used in some languages, but I'm not sure if there were any problems with that one.

[iseulde]

Fix from TinyMCE: ​https://github.com/tinymce/tinymce/commit/1696e387e818d9c3f4e2e2d8871f0218208a0212
Let's consider it for 4.6.1 with the other TinyMCE fixes, otherwise it will be for 4.7.

[iseulde]

Please note the comments from spocke: ​https://github.com/tinymce/tinymce/issues/2859#issuecomment-241992872

[nathanwells]

Replying to iseulde:

Thanks for the report and welcome to Trac! Is this problem new to 4.6?

No, it is not new - started in Wordpress 4.3.x but since the but was only posted in the TinyMCE GitHub I thought it would be good to bring attention to it here as well since it impacts Wordpress (and since the previous time this happened, it was dealt with here). Also it is not new in the fact that it was fixed once already, but now the code has regressed and needs to be patched again.

We are also using that character for the text patterns, but it is carefully inserted and removed again. We are not removing just any character. We should maybe use \uFEFF though if the other is used in some languages, but I'm not sure if there were any problems with that one.

I would think using \uFEFF could be a good option for us (the Khmer language). Based on Wikipedia, some languages used to use \uFEFF as a type of zero-width space but that has now been depreciated, and it is used exclusively as the BOM ​Wikipedia:Word Joiner and ​Unicode FAQ

Appreciate your time on this!

[nathanwells]

Just saw the ​patch - wonderful! Thank you @iseulde and spocke!

[azaozz]

IMHO we should patch this in 4.6.1. That means we will have to push an update to TinyMCE which might cause some "caching jitters" on a few sites for a day, but having the editor strip a character that was typed by the users is a "high impact" bug.

Thinking best would be to backport the above fix to the current TinyMCE version in 4.6. That way we will minimize any possibilities of breaking something else in 4.6.1.

[azaozz]

Replying to azaozz:

Turns out this is not that simple. Changing the zero-width space char (placeholder) in TinyMCE requires other changes and fixes, most for IE. See ​https://github.com/tinymce/tinymce/commit/16750c20fdba279ff98f0e45aea5e859b1384e27.

In that terms we would have to backport several changes that are (mostly) untested in WordPress. Not sure this is good for a dot release. This would have been very nice to fix but is not a regression from 4.5. Seems better to leave it for 4.7.

[azaozz]

This is fixed upstream and will be fixed in WordPress when we update TinyMCE in 4.7.

[jorbin]

@azaozz Do you already have a plan for when you are going to update TinyMCE in 4.7?

[jorbin]

I've opened #38081 to update TinyMCE which should resolve this issue.

[azaozz]

In 38773:

TiinyMCE: update to 4.4.3, changelog: ​https://www.tinymce.com/docs/changelog/#version443-september12016

Fixes #38081, #38245, #37507, #37808 and #38000.