UTF-8 strings are sometimes cut in the middle of a character

[nbachiyski]

Using substr on UTF-8 strings can cause some characters to be cut on the middle, because substr counts bytes, but in UTF-8 a character can be more than one byte.

Here is a patch, which:

• Defines mb_strcut in compat.php}} for the users, who don't have the {{{mb_string extension.
• Introduces a new wp_html_excerpt function, which uses mb_strcut and works well with html strings: counts entities as one character (& isn't 4 chars) and strips tags.

There are some tests for the two functions:

• ​_mb_strcut
• ​wp_html_excerpt (in the end of the file)

(In [7140]) Multi-byte character safe excerpting from nbachiyski. fixes #6077

Oh, I was misled by ​the html_entity_decode manual, which says:

Version   Description
5.0.0     Support for multi-byte character sets was added.

I didn't see above this message was written that most of the encodings we need are supported by 4.3.0. So, let's add the encoding then.

The manual says:

Any other character sets are not recognized
 and ISO-8859-1 will be used instead.

This is good thing but it outputs warnings in that case, so I updated the patch just to add "@" anyway.

According to the PHP user notes, html_entity_decode() has a bug with UTF-8.
Maybe we shoud create substitute function?

bug should be reproduced with this code before PHP 5.0.1:

echo html_entity_decode('€', ENT_QUOTES, 'UTF-8');

We can just drop the entity decoding part. Yes -- the excerpt could be a couple of characters shorter than the specified length, but that's how it has worked up to now and nobody complained.

Here is a patch, which removes entity decoding. Documentation and test are also updated.

(In [7190]) Remove the entity decoding and recoding from wp_html_excerpt. Fixes #6077 props nbachiyski.

It looks like this bug has resurfaced in the RSS2 comments feed at:

​http://www.arnaudmontebourg.fr/?feed=rss2

I don't run that site, but this was reported as a bug in a software project I maintain:

​https://code.google.com/p/feedparser/issues/detail?id=306

Replying to kurtmckee:

This ticket was closed on a completed milestone. Please open a new one.