Codepage issue with the wp.org Credits API

[demetris]

I was playing with my profile page at wp.org and changed the Name field from:

demetris

to:

demetris (Δημήτρης Κίκιζας)

What the API returns for that is:

demetris (???????? ???????)

It seems the API returns its results in ISO 8859-1. Can we change that to UTF-8?

[westi]

I spent a good amount of time looking into this and it is not an easy fix at the moment.

We have a mix of UTF8 and latin1 tables on WP.org and the api is serving data from a latin1 table.

We can revisit this later but for now I have changed the api to return the user name when we detect this.

[bi0xid]

I have solved it changing the tables to UTF-8 and doing a "Search and Replace".

Attached FYI.

[bi0xid]

Replace LATIN1 -> UTF-8

[dd32]

could ​mb_convert_encoding() be used here at all?

closed #17915 as duplicate

[westi]

Replying to bi0xid:

I have solved it changing the tables to UTF-8 and doing a "Search and Replace".

Attached FYI.

While this might look like it works in only handles a subset of common case issues and doesn't really resolve the underlying issue.

We have a lot of tables and a lot of data - there have been a lot of accounts created over time on WP.org for the forums and we need to do this in a careful and reliable manner.

This is not the only manifestation of the issue and we need to work carefully to resolve them all - hopefully that will happen soon.

[bi0xid]

Replying to westi:

While this might look like it works in only handles a subset of common case issues and doesn't really resolve the underlying issue.

I know. It's just a patch for everyone who needs a quick solve while we find a good solution. These are the common cases for Spanish transformations.

[nacin]

Since some bad data was still being returned via the API due to HTML entities, there is now also a remove_accents() call on the names.

[samuelsidler]

Migrated to ​#meta128.