Sorting Greek characters

[beerallica]

There's an abnormal behaviour in alphabetically sorted lists containing terms in Greek. The expected order would be Latin characters on top, then Greek; two consecutive lists. However, the Greek list is actually further split into two other lists, one with terms starting with an upper case character and the other with terms starting with lower case. So it's Latin Aa-Zz, then Greek Α-Ω, then Greek α-ω. A real-world example is a tag cloud. See this for example: ​http://www.tastefull.gr/tagcloud

[beerallica]

Update: The lists are five, not three:

1. Latin case-insensitive
2. Uppercase accented Greek
3. Uppercase non-accented Greek
4. Lowercase accented Greek
5. Lowercase non-accented Greek

The database collation is utf8mb4_unicode_ci and phpmyadmin sorts terms OK (e.g. Latin case-insensitive, then Greek case/accent-insensitive).

[desrosj]

Hi @beerallica,

My apologies that this ticket took so long to receive a response. I found this ticket while scrubbing a report for tickets missing a milestone.

Since it's been a substantial amount of time, I wanted to ask if you were still experiencing this in the latest version of WordPress (currently 6.6.1).

If so, could you provide a set of values as an example to see the behavior? Also, how are you sorting these values? Is this an alphabetic sort in a WP_Query? Something else? If you are a developer and comfortable in providing one, a unit test showing the bug would also be helpful.

Greek terms in a tag cloud come out in five groups instead of two. The reporter described it as Latin Aa-Zz, then Greek Α-Ω, then Greek α-ω, and later refined it to five groups: Latin case-insensitive, uppercase accented Greek, uppercase non-accented Greek, lowercase accented Greek and lowercase non-accented Greek.

wp_generate_tag_cloud() re-sorts the terms in PHP after the database has already sorted them. That second sort goes through _wp_object_name_sort_cb(), which compared with strnatcasecmp(). That function folds case for ASCII only and ignores accents entirely, so for anything non-ASCII the comparison is decided by raw byte value, and one Greek alphabet becomes four byte ranges:

uppercase accented   U+0386-U+038F   CE86-CE8F
uppercase plain      U+0391-U+03A9   CE91-CE9F
lowercase accented   U+03AC-U+03CE   CEAC-CEAF
lowercase plain      U+03B1-U+03C9   CEB1-CF89

Those four ranges stay distinct and sort in exactly that order, which is the reported symptom. It also explains the part of the report that was hardest to account for: the same terms sorted correctly in phpMyAdmin, because the database collation is doing the right thing and the PHP re-sort is throwing that ordering away.

The fix builds a sort key by folding the name to lowercase and stripping combining marks, so case and accent variants of the same letter share a key and sort together. Two details worth calling out:

• The key is deliberately left in decomposed form. Recomposing after stripping the marks puts the accents straight back, so the fold is undone.
• A deterministic strnatcmp() tie-break on the original name runs when the folded keys are equal. uasort() is not stable, so without a tie-break the relative order of variants would be undefined and untestable.

mbstring and intl are not required by WordPress and neither is polyfilled for these functions, so both folds apply only when the extension is present, following the function_exists() pattern already used in wp-includes/class-wp-query.php:

$strtolower = function_exists( 'mb_strtolower' ) ? 'mb_strtolower' : 'strtolower';

With neither extension available the ordering is no worse than before: the plain strtolower() fallback still merges uppercase and lowercase forms, covering the larger half of the report.

Two additional notes:

• The callback also returned the result of strnatcasecmp() directly. It now returns an int. This is the same class of issue that was fixed for an equivalent comparison callback in wp-includes/post.php, where returning a bool triggered a deprecation notice on PHP 8. It is mentioned here because the touched function is the one in the fix, not as a claim about any other callback.
• No change was made to remove_accents(). It has no Greek entries in its character map and no el_GR branch in its locale rules, so it is the wrong tool for this and adding a locale branch there would be a much larger change than this ticket needs.

Tests were added to the existing tag cloud test file rather than a new one, since that file is where wp_generate_tag_cloud() coverage already lives. They drive the public API, and the ASCII test passes both with and without the fix, so the change is pinned not to alter existing ordering.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude
Model(s): Sonnet
Used for: test cases, code review