Encoding errors in AJAX responses

[sergey.s.betke@…]

When the option default_charset in php.ini (for example - windows-1251) other than blog encoding (for example - UTF-8), whe have troubles.

WordPress version

Latest WordPress files - 3.3-RC1-19537

browser

MS IE9

Reproducing the problems

• set in php.ini (or in htaccess) default_charset to windows-1251 (or any other, not UTF-8 in this example)
• set wordpress option blog_charset to UTF-8
• go to /wp-admin/edit-comments.php?comment_status=trash for example (we have troubles not just here)
• restore one comment, and look to comments count after a few seconds. I see 0 элементов instead of 0 комментариев.
• or press Reply at /wp-admin/edit-comments.php?comment_status=approved, type тест (test in russian language), and Ok. I see: Отправлен 05.12.2011 РІ 4:50 РїРї | Р’ ответ автору webmaster@….

тест

Отклонить | Ответить | Свойства | Изменить | Спам | Удалить

plugins and themes

This trouble is exist on default wordpress installation, without any plugins, on default theme

Test environment

• IIS 7
• PHP 5.3.6
• MySQL 5.1
• proxy - MS ISA Server
• Test server - ​http://test.www.novgaro.ru, account - testuser, password - testuser. (You can login to this test server).

[sergey.s.betke@…]

Google Chrome - the same results

[sergey.s.betke@…]

+ nocache headers...

[SergeyBiryukov]

Related: #10817, #11034

[SergeyBiryukov]

Another way to reproduce (from #10817):

1. Set default_charset parameter in php.ini to 'iso-8859-2'.
2. Make sure the blog charset in Settings → Reading is UTF-8.
3. Go to Posts → Categories, create a new category: "Árvíztűrő tükörfúrógép".
4. The AJAX response is wrong: 19448.response.png​.
5. If default_charset is set to 'windows-1251', the response doesn't appear at all due to an error:

   Timestamp: 07.08.2012 14:16:41
   Error: not well-formed
   Source File: http://trunk.wordpress/wp-admin/admin-ajax.php
   Line: 1, Column: 903
   Source Code:
   <?xml version='1.0' standalone='yes'?><wp_ajax><response action='add-tag_0'><taxonomy id='0' position='1'><response_data><![CDATA[]]></response_data><supplemental><parents><![CDATA[<tr id="tag-149"><th scope="row" class="check-column"><label class="screen-reader-text" for="cb-select-149">Select ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p</label><input type="checkbox" name="delete_tags[]" value="149" id="cb-select-149" /></th><td class='name column-name'><strong><a class="row-title" href="http://trunk.wordpress/wp-admin/edit-tags.php?action=edit&#038;taxonomy=category&#038;tag_ID=149&#038;post_type=post" title="Редактировать &laquo;ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p&raquo;">ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p</a></strong><br /><div class="row-actions"><span class='edit'><a href="http://trunk.wordpress/wp-admin/edit-tags.php?action=edit&#038;taxonomy=category&#038;tag_ID=149&#038;post_type=post">Р￿зменить</a> | </span><span class='inline hide-if-no-js'><a href="#" class="editinline">Свойства</a> | </span><span class='delete'><a class='delete-tag' href='edit-tags.php?action=delete&amp;taxonomy=category&amp;tag_ID=149&amp;_wpnonce=23ee944760'>Удалить</a> | </span><span class='view'><a href="http://trunk.wordpress/category/arvizturo-tukorfurogep/">Перейти</a></span></div><div class="hidden" id="inline_149"><div class="name">ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p</div><div class="slug">arvizturo-tukorfurogep</div><div class="parent">0</div></div></td><td class='description column-description'></td><td class='slug column-slug'>arvizturo-tukorfurogep</td><td class='posts column-posts'><a href='edit.php?category_name=arvizturo-tukorfurogep'>0</a></td></tr>]]></parents><noparents><![CDATA[<tr id="tag-149" class="alternate"><th scope="row" class="check-column"><label class="screen-reader-text" for="cb-select-149">Select ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p</label><input type="checkbox" name="delete_tags[]" value="149" id="cb-select-149" /></th><td class='name column-name'><strong><a class="row-title" href="http://trunk.wordpress/wp-admin/edit-tags.php?action=edit&#038;taxonomy=category&#038;tag_ID=149&#038;post_type=post" title="Редактировать &laquo;ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p&raquo;">ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p</a></strong><br /><div class="row-actions"><span class='edit'><a href="http://trunk.wordpress/wp-admin/edit-tags.php?action=edit&#038;taxonomy=category&#038;tag_ID=149&#038;post_type=post">Р�зменить</a> | </span><span class='inline hide-if-no-js'><a href="#" class="editinline">Свойства</a> | </span><span class='delete'><a class='delete-tag' href='edit-tags.php?action=delete&amp;taxonomy=category&amp;tag_ID=149&amp;_wpnonce=23ee944760'>Удалить</a> | </span><span class='view'><a href="http://trunk.wordpress/category/arvizturo-tukorfurogep/">Перейти</a></span></div><div class="hidden" id="inline_149"><div class="name">ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p</div><div class="slug">arvizturo-tukorfurogep</div><div class="parent">0</div></div></td><td class='description column-description'></td><td class='slug column-slug'>arvizturo-tukorfurogep</td><td class='posts column-posts'><a href='edit.php?category_name=arvizturo-tukorfurogep'>0</a></td></tr>]]></noparents></supplemental></taxonomy></response><response action='add-tag_0'><term id='0' position='0'><response_data><![CDATA[]]></response_data><supplemental><term_id><![CDATA[149]]></term_id><name><![CDATA[ГЃrvГ­ztЕ±rЕ‘ tГјkГ¶rfГєrГіgГ©p]]></name><slug><![CDATA[arvizturo-tukorfurogep]]></slug><term_group><![CDATA[0]]></term_group><term_taxonomy_id><![CDATA[166]]></term_taxonomy_id><taxonomy><![CDATA[category]]></taxonomy><description><![CDATA[]]></description><parent><![CDATA[0]]></parent><count><![CDATA[0]]></count></supplemental></term></response></wp_ajax>
   

   Timestamp: 07.08.2012 14:16:41
   Error: TypeError: res.responses[0] is undefined
   Source File: http://trunk.wordpress/wp-admin/js/tags.dev.js?ver=3.5-alpha-21466
   Line: 46
   

AddDefaultCharset utf-8 in .htaccess doesn't help.

19448.2.diff​ is the refreshed patch. Not sure if nocache_headers() and Content-Length are necessary though.
19448.charset-only.diff​ just fixes the charset issue.

[azaozz]

19448.2.diff looks good. Wondering if/how would this work by using json instead of xml.

[SergeyBiryukov]

Related: #4794, #19454

[kurtpayne]

Unit test

[kurtpayne]

Unit test added. Patch looks good.

[ryan]

​[UT972]

[ryan]

In [21528]:

Specify the charset in WP_Ajax_Response::send(). Props sergey.s.betke@…, SergeyBiryukov. fixes #19448

[ryan]

Should send() even do the header() call if DOING_AJAX? admin-ajax.php probably already took care of it.

[SergeyBiryukov]

Replying to ryan:

Should send() even do the header() call if DOING_AJAX? admin-ajax.php probably already took care of it.

The header() call in admin-ajax.php sets Content-Type: text/html:
​http://core.trac.wordpress.org/browser/tags/3.4.1/wp-admin/admin-ajax.php#L32

Removing the one in send() would break XML responses (e.g. when adding a tag).