add_metadata() Fails to Store Serialized Values as BINARY

[miqrogroove]

Symptoms:

WordPress stores corrupt values in post_metadata if there are any non-UTF-8 bytes in the meta_value.

Steps to reproduce:

Call add_metadata() with non-UTF-8 values such as a latin-1 copyright char.

Even though the serialized string goes through prepare() before the query, MySQL is required to truncate the invalid value being assigned to the meta_value field. The result is that the stored value can never be un-serialized.

This behavior can also be replicated by trying to inject CHAR(169) into any UTF-8 table query.

[miqrogroove]

After giving this some thought, I've come to the conclusions:

1. Post metas are not binary-safe.

2. There is no way to make them binary-safe without converting all existing values to a proper binary column.

[Denis-de-Bernardy]

similar bugs referenced here: #9549

[dd32]

Shifting to Future release pending patch, Too close to release to be changing anything related to the Schema anyway.

[bpetty]

As the MySQL connection and encoding is defined through the global DB_COLLATE and DB_CHARSET definitions on installation, and I don't believe there's ever going to be the possibility that metadata values will ever be re-encoded to a new binary-safe encoding on upgrade, I just don't think this will ever be supported.

If you absolutely need to store non-UTF8 encoded strings or binary data, the best option I can think of is to do so in a custom table with your explicit schema, or somewhere other than in the DB. Otherwise it should be expected that your strings are encoded with whatever DB_CHARSET was in use on installation.

[nacin]

Yeah, I don't see this as something we're going to do.

[miqrogroove]

Why not? BINARY is compatible with all encodings. The upgrade would be easy. Trivial even. I hear so many WordPress developers complain about this! What a shame.