IPTC IIM IRB character encoding (UTF-8) misinterpreted as ISO-8859-1

[kallewangstedt]

When uploading a JPEG image with IPTC tags in the EXIF that are encoded as UTF-8, the tags are misinterpreted as ISO-8859-1 when they get extracted into the Title and Description fields in swfupload.

This occurs regardless of the encoding set in WP options (wich are UTF-8 by default).

[Denis-de-Bernardy]

a sample image might help

[hakre]

In a check yesterday I encountered ISO-encoded files in SWF-Uploader. I guess this is the cause of evil. A core-dev should ensure that all files are UTF-8 encoded. I make a try if that is patchable.

[hakre]

utf-8 fix

[hakre]

Please test

[Denis-de-Bernardy]

This actually fixed things on your end? :-)

[Denis-de-Bernardy]

Replying to hakre:

Please test

Would love to, but wouldn't know how... or where to find the needed test data.

[Denis-de-Bernardy]

see also #9413

[hakre]

I could not test because I do not have such a jpeg file with that exif data. but maybe this is related since with encodings the chain of data must stay intact.

[Denis-de-Bernardy]

hehe, sounds like one of those bugs that won't get fixed from lack of valid test cases. :D

[hakre]

+1 having the files consistently encoded anyway. they are loaded in from utf-8 output (normally) and therefore should be utf-8 encoded as well.

[demetris]

Bug confirmed. I see exactly what kallewangstedt describes.

hakre’s patch does not help. (I think the patched file is not used at all, as it is merged and minified into swfupload-all.js, and, in any case, the patch just changes one character in a comment block.)

I attached a picture, for anyone to see how the characters appear (utf-8 read as iso-8859-1).

The picture is in Picasa Web, if anyone wants to test:

​http://picasaweb.google.gr/demetris.pics/2

The ITPC field I used for testing is: caption-abstract. Notice that the text appears fine in Picasa Web.

[demetris]

Sample JPEG file with trilingual UTF-8 text in the ITPC caption-abstract field

[demetris]

Screenshot of Flash uploader with UTF-8 text in ITPC field read as ISO-8859-1

[Denis-de-Bernardy]

punting to future pending a patch, now that we've test data.

[hakre]

iptcparse() is used to gather iptc data. it is assumed that the iptc data is ISO-8859-1 encoded and is converted to utf8 by utf8_encode() then.

this is not a swfupload issue. this is a encoding issue based on the handling within wordpress.

so the question is: how is the iptc encoding done? is it marked? at least a test for having utf8 data in there seems reasonable to me. if it is already valid utf8 then good to go. if not, the latin-1 can be assumed and encoded into utf8 (as it is done now).

[Denis-de-Bernardy]

@hakre: would you like charset issues to be automatically assigned to you?

[hakre]

Fix. Patch does check for having utf-8 previous encoding to it. And thats it (for now). Tested with the provided test-image and it does the job. Copy and Paste from the description textbox: "This is a comment. / Это комментарий. / Βλέπετε ένα σχόλιο.".

So this does the job without following the specs. For those I dropped a link in the docblock. The subject is IPTC IIM IRB (IPTC Information Interchange Model Image Resource Blocks) just for note in case this issue needs more attention in the future.

@Denis: you can do so, let's look how it goes.

[Denis-de-Bernardy]

done. you left a var_dump() in your patch.

[hakre]

existing UTF8 encoding of IPTC blocks preserved

[hakre]

@var_dump() - it's getting too late over here. removed it from the patch.

[Denis-de-Bernardy]

punting per IRC discussion

[hakre]

can not find that discussion? any details on current state? any problems with the patch?

[dd32]

IRC discussion was about punting any non-critical bugs from 2.8 due to pending release.

[hakre]

Then I put this on the list for 2.8.2, because I assume 2.8.1 should be for criticals as well.

[hakre]

related: #9417

[hakre]

related: #6412

[yoavf]

Double tested :)

[hakre]

I just got some more files (from various systems) and some code by a coder I met on a conf in september. I will test this within the current bughunt. I'm pretty shure this will improve the topic.

[westi]

Move to 3.0 for now.

[hakre]

Related: #7580

[hakre]

Reference: #11547

[hakre]

Related: #11417

[miqrogroove]

#12095

[hakre]

from php.ini-development (as of PHP 5.3.5), see the EXIF section:

[exif]
; Exif UNICODE user comments are handled as UCS-2BE/UCS-2LE and JIS as JIS.
; With mbstring support this will automatically be converted into the encoding
; given by corresponding encode setting. When empty mbstring.internal_encoding
; is used. For the decode settings you can distinguish between motorola and
; intel byte order. A decode setting cannot be empty.
; ​http://php.net/exif.encode-unicode
;exif.encode_unicode = ISO-8859-15

; ​http://php.net/exif.decode-unicode-motorola
;exif.decode_unicode_motorola = UCS-2BE

; ​http://php.net/exif.decode-unicode-intel
;exif.decode_unicode_intel = UCS-2LE

; ​http://php.net/exif.encode-jis
;exif.encode_jis =

; ​http://php.net/exif.decode-jis-motorola
;exif.decode_jis_motorola = JIS

; ​http://php.net/exif.decode-jis-intel
;exif.decode_jis_intel = JIS

I just stumbeled over that information, it might be that this might be related to a configuration issue as well.

[SergeyBiryukov]

Looks like the tags are double-encoded if they are already in UTF-8. The approach from 9417.2.patch​ works for me. Refreshed for current trunk.

[nacin]

We need unit tests here. Anyone want to tackle it now? Looks good though. Probably long past time to punt it.

[westi]

Replying to nacin:

We need unit tests here. Anyone want to tackle it now? Looks good though. Probably long past time to punt it.

Still no unit tests punting.

[SergeyBiryukov]

​[UT665]

[SergeyBiryukov]

Closed #20408 as a duplicate.

[SergeyBiryukov]

Refreshed

[SergeyBiryukov]

#21903 was marked as a duplicate.

[nacin]

The patch on #21903 tries to use the IPTC identifier for UTF-8. If that is consistently accurate, I prefer it over this. Some research on the standard would be great — otherwise, this seems fine.

[SergeyBiryukov]

Replying to nacin:

The patch on #21903 tries to use the IPTC identifier for UTF-8. If that is consistently accurate, I prefer it over this.

From my testing, $iptc['1#090'] marker may not always be present: ticket:20408:3.

[chenxing]

Replying to SergeyBiryukov:

Replying to nacin:

The patch on #21903 tries to use the IPTC identifier for UTF-8. If that is consistently accurate, I prefer it over this.

From my testing, $iptc['1#090'] marker may not always be present: ticket:20408:3.

I don't know if there is a reliable source. I got it from here: ​http://php.net/manual/en/function.iptcparse.php#105025

I tried to Google for a reliable source but with no luck...

[nacin]

In [21905]:

Avoid mangling UTF-8 strings that may be present in image metadata. props SergeyBiryukov for the unit tests ​[UT665]. fixes #9417.