Smart quote apostrophe ’ results in a permalink URL with %e2%80%99

[foolswisdom]

Smart quote apostrophe ’ results in a permalink URL (slug) with %e2%80%99

ENV: WP trunk r4915

smart quote apostrophe ’
Mac shortcut: Using Shift - Option - ]

ADDITIONAL DETAILS
My guess is that a solution should identify allowed characters, translated to hyphen -, and strip all the others.

[rob1n]

This is proper behavior. The curly quote isn't plaintext -- it's a symbol and has to be translated. The same is for other UTF-8 symbols such as Chinese characters (some other bug was about that) -- they are and should be turned into URL-safe entities.

[markjaquith]

Solution would have to deal with this case specifically. Note that the URL, while ugly, is functional. Also note that in 2.1, people should be able to edit their post slug and have the old one redirect to the current one.

[jhodgdon]

Just a little clarification. The function being used to create the slug is sanitize_title_with_dashes in wp-includes/formatting.php

The sequence of events is currently:

1) Post title becomes the slug candidate

2) Accents are removed (replaced by un-accented letters)

3) Characters that still look like they are UTF-8 are encoded with utf8_uri_encode into octets (%e2, etc.) (this is what is creating the reported behavior)

4) HTML entities and any character except letters, numbers, underscores, spaces, octets, and hyphens are removed (this is where other punctuation is removed)

5) Spaces are turned into hyphens, and whole thing is lower-cased

So... to fix this, would have to add step 2.5:

2.5: Translate into hyphens, or remove (more consistent with what happens to other punctuation), a specific list of special (but common) punctuation characters.

Questions:

a) Is this worth doing, considering that the current behavior makes a usable slug, and that you can always edit your slug by hand if you want to?

b) If it is worth doing, what should the list of special punctuation characters be, and should they be removed or translated into hyphens?

[drmike]

Issue exists over in wp.com land as well:

​http://en.forums.wordpress.com/topic.php?id=9645

[drmike]

Also why not just strip it out?

[rob1n]

Well, we strip *regular* quotes out, but not fancy quotes. I think this is really not going to be fixed easily -- we can strip out UTF-8 quotes, but what about other encodings?

[pishmishy]

Please can you leave a comment explaining why you've closed the ticket.

[noel]

This patch should fix the problem - we were treating all unicode as equal - when we should have been defining the different categories and removing the unicode characters relevant to punctuation, etc.

This patch simply attaches onto the other sanitize_title functions and will probably need to be integrated more fully in the future. As for now, it works great for me on all the test cases I threw at it.

In the future, when all browsers support full unicode characters in the URL shouldn't we not be converting them at all? ;)

[noel]

Permalink filter for unicode punctuation.

[ryan]

Any changes to the sanitizer will lead to 404s for slugs made with the old sanitizer.

[noel]

I'll get that sorted out and resubmit a patch.

[noel]

Unicode fixes that do not produce a 404.

[ryan]

This will cause old permalinks that have %e2%80%99 in them to 404.

[westi]

As with #3206 i am pushing this out we need a single solution to the whole mess ;-)

[Denis-de-Bernardy]

Merging into #9591.

[scribu]

It's better to have remaining issues highlighted through open tickets.