XMLRPC api stripping leading angle brackets

[midrangeman]

When posting from an application that uses the wordpress xmlrpc api, leading angle brackets are being stripped from the post.

The following was posted from the Wordpress iPhone app, but similar results have been observed with Flickr posting and other apps...

p>a href="!http://beta.geekyramblings.org/wp-content/uploads/2008/09/p-640-480-2fd9116d-
5703-4e8b-b72c-2605beab02b5.jpeg">img src="!http://beta.geekyramblings.org/wp-content/uploads
/2008/09/p-640-480-2fd9116d-5703-4e8b-b72c-2605beab02b5.jpeg" alt="" width="225" height="300" 
class="alignnone size-full wp-image-364" />/a>/p>

This seems to have manifest in version 2.6.2.

[josephscott]

Are you able to sniff the traffic going to WP? I want to make sure that it wasn't removed by the iphone app before being sent to WP. I'll also see if I can duplicate this in the iphone simulator.

[midrangeman]

I don't think it's related to iPhone, as my wife encountered the same behaviour when she submitted a post from Flickr or Scribefire.

None the less, I'll fire up wireshark and see if I can't capture some traffic to verify it.

[midrangeman]

I'm not 100% sure this is what you are looking for, but I've attached what wireshark captured.

[DD32]

Wireshark capture of posting from iPhone.

You didnt forget to remove your credentials from that did you?

[midrangeman]

It's a test blog, so I'm not super worried. I've redacted the information anyways.

[alykhan2]

I'm having the exact same problem with MarsEdit and Ecto.

I just posted to my blog with Ecto and this was the end result

​http://www.airlinenation.com/blog/2008/09/24/airline-news/indian-carrier-kingfisher-scaling-back.htm

---

pIndian carrier Kingfisher Airlines is returning some of its leased aircraft and has started to retrench some workers in an effort to cut costs and stem its losses./p
pThe company has also identified 300 employees who will be retrenched shortly but these will be voluntary redundancies, says the spokesman, adding that the retrenched workers will get two months pay for every year of service/p
pThe airline is handing back some of its leased aircraft to “achieve cost savings and rationalisation and operational efficiencies”, says the spokesman, adding that this has taken on greater urgency because of “the ongoing turbulence faced by the aviation industry”./p
pAbout 1 month ago Kingfisher started international services to London, Heathrow airport./p

---

As you can see all the <> tags were stripped off.

[ricolas]

I met the same problems, When I published blogs from Microsoft Word 2007 and Windows Live Writer, all the <> tags were stripped off.

[josephscott]

I have tried to confirm this and so far no luck. Both 2.6.2 release and -trunk added posts with a leading < and it came across fine. I kept the post content very basic:

<p>Hello World!</p>

The entire XML-RPC request went across the wire as :

<?xml version="1.0"?>
<methodCall>
<methodName>metaWeblog.newPost</methodName>
<params>
<param><value><int>1</int></value></param>
<param><value><string>******username********</string></value></param>
<param><value><string>******password********</string></value></param>
<param><value><struct>
  <member><name>title</name><value><string>script angle bracket post</string></value></member>
  <member><name>description</name><value><string>&lt;p&gt;this is a test with an angle bracket&lt;/p&gt;</string></value></member>
</struct></value></param>
<param><value><boolean>0</boolean></value></param>
</params></methodCall>

It then showed in WP normally, with the paragraph tags (including brackets) intact.

I'm happy to get this sorted out, but the first thing is figuring out how to recreate the problem. Anyone who is having this problem please let me know, I'm usually on #wordpress-dev on FreeNode (username - josephscott).

[josephscott]

After looking around some more, this appears to be a PHP level problem with the XML parser:

​http://bugs.php.net/bug.php?id=46138

​http://bugs.php.net/bug.php?id=45996

[alykhan2]

Replying to alykhan2:

I'm having the exact same problem with MarsEdit and Ecto.

I just posted to my blog with Ecto and this was the end result

​http://www.airlinenation.com/blog/2008/09/24/airline-news/indian-carrier-kingfisher-scaling-back.htm

---

pIndian carrier Kingfisher Airlines is returning some of its leased aircraft and has started to retrench some workers in an effort to cut costs and stem its losses./p
pThe company has also identified 300 employees who will be retrenched shortly but these will be voluntary redundancies, says the spokesman, adding that the retrenched workers will get two months pay for every year of service/p
pThe airline is handing back some of its leased aircraft to “achieve cost savings and rationalisation and operational efficiencies”, says the spokesman, adding that this has taken on greater urgency because of “the ongoing turbulence faced by the aviation industry”./p
pAbout 1 month ago Kingfisher started international services to London, Heathrow airport./p

---

As you can see all the <> tags were stripped off.

I'm running PHP 5.2.6

DOM/XML API Version 20031129
libxml Version 2.7.1
libXML support active
libXML Version 2.7.1
libXML streams enabled
.

Strangely this problem only started after I upgraded to 2.6.2. (I was on 2.5.x before)

[josephscott]

I wonder if this is specific to the libxml version then. Here's the details on what I tested against:

PHP - 5.2.6
libxml - 2.6.16
dom/xml api version - 20031129

So perhaps there's something different about libxml 2.7.x that is causing the problems?

[jeppe]

As a temporary workaround you can compile PHP against expat with --with-libexpat-dir=/usr to get this working.

[westi]

Reading the referenced PHP bugs and the libxml mailing list discussion it sounds like the PHP extension is broken with newer xml because it was misusing the libxml code inorder to try and simulate how it used to work with expat.

I wonder if we can switch IXR easily to another parser if one is available and only fall back to the PHP4 compatible one if required.

[josephscott]

I'm closing this ticket, it appears to be a problem with a specific version of libxml that was fixed in libxml 2.7.2.

[midrangeman]

I upgraded to libxml2 2.7.2 (testing RPM) on my FC8 system and the problem still exists.

[josephscott]

Replying to midrangeman:

I upgraded to libxml2 2.7.2 (testing RPM) on my FC8 system and the problem still exists.

Are you able to roll back to a pre 2.7.x libxml and see if that fixes things.

[Sixes]

It seems that this is a bug in libxml2's emulation of the expat library from version 2.7.1 onwards and the fact that php will use the libxml2 library for expat emulation if the expat library is not compiled in. Indeed, php 5.2.6 shows libexpat as deprecated and the Fedora 9 install of php does not include it.

It seems that everyone is waiting for everyone else to fix this now. Possibly the simplest way forward is for Wordpress to stop using the XML Parser library (xml_parser_create() and friends) and switch to the experimental but working XML-RPC library (xmlrpc_server_create() etc.). This should avoid the problem as presumably it uses native libxml2.

The above is hypothesis. I'm going to try testing it now.

[midrangeman]

Replying to josephscott:

Are you able to roll back to a pre 2.7.x libxml and see if that
fixes things.

Not sure ... the dependencies in my Fedora 8 system might preclude it, but I'll see what I can do.

[josephscott]

Replying to Sixes:

It seems that everyone is waiting for everyone else to fix this now. Possibly the simplest way forward is for Wordpress to stop using the XML Parser library (xml_parser_create() and friends) and switch to the experimental but working XML-RPC library (xmlrpc_server_create() etc.). This should avoid the problem as presumably it uses native libxml2.

The above is hypothesis. I'm going to try testing it now.

If we can come up with a reliable replacement approach I'd be fine to do that. So far I haven't been able to reproduce this on any of my systems, which are probably all using older versions of libxml.

[midrangeman]

Replying to josephscott:

If we can come up with a reliable replacement approach I'd be
fine to do that. So far I haven't been able to reproduce
this on any of my systems, which are probably all using
older versions of libxml.

If need be, I can provide an account on my system where this can be reproduced.

Contact me david at midrange.com.

[redsweater]

Data point from one of my customers:

​http://www.red-sweater.com/forums/viewtopic.php?pid=5286#p5286

Suggesting that the xmlrpc.php file from 2.6.5 exhibits the problem, but the 2.6.3 one does not?

[westi]

Replying to redsweater:

Data point from one of my customers:

​http://www.red-sweater.com/forums/viewtopic.php?pid=5286#p5286

Suggesting that the xmlrpc.php file from 2.6.5 exhibits the problem, but the 2.6.3 one does not?

I would be surprised if that was true considering the differences between the two:

​Changes between 2.6.3 and 2.6.5 xmlrpc.php

2.6.3 is rev [8445] of xmlrpc.php
2.6.5 is rev [9747] of xmlrpc.php

[redsweater]

Hmm - this makes me wonder if something could be corrupting xmlrpc.php files. Spam infestation or a bad plugin somehow?

The customer who made the claim abut 2.6.3 and 2.6.5 has later confirmed that putting the "factory" 2.6.3 xmlrpc.php up also seems to address the problem.

[midrangeman]

Replying to redsweater:

Hmm - this makes me wonder if something could be corrupting xmlrpc.php files.
Spam infestation or a bad plugin somehow?

The problem I have has persisted through multiple WP versions ... including 2.6.3.

[mmcclusky]

Having this same issue in 2.7 using MarsEdit -- just started after upgrading WP.

Attempting to downgrade to 2.6.1 xmlrpc.php didn't solve the issue. Server admins report no changes to webhost configuration.

[mmcclusky]

Sorry, also forgot to mention I'm seeing this same behavior from Flickr uploads as well.

Replying to mmcclusky:

Having this same issue in 2.7 using MarsEdit -- just started after upgrading WP.

Attempting to downgrade to 2.6.1 xmlrpc.php didn't solve the issue. Server admins report no changes to webhost configuration.

[josephscott]

I found that this problem doesn't happen when using libxml2 <= 2.6.30

It's also been reported that this bug doesn't happen if PHP using expat for XML functions instead of libxml2.

[bemmu]

I started having this problem when I moved from a server with libxml 2.6.30 to one with libxml 2.6.31 (have no write privilege so cannot downgrade libraries). The problem-causing calls appear to happen in wp-includes/class-IXR.php in the function parse().

[isharasamantha]

This is the workaround patch for wordpress users.If you can not downgrade libxml version or can not wait libxml2 fix it later.

​http://blog.hoofoo.net/2009/01/14/wordpress-patch-for-problamatic-libxml2-version/

[jcanady]

Replying to isharasamantha:

This is the workaround patch for wordpress users.If you can not downgrade libxml version or can not wait libxml2 fix it later.

​http://blog.hoofoo.net/2009/01/14/wordpress-patch-for-problamatic-libxml2-version/

I've created an actual .diff patchfile for these changes. The patch is against the 2.7 tag from SVN, so should work with the 2.7 stable release tarballs.

​http://joncanady.com/2009/01/wordpress-xmlrpc-angle-bracket-fix/

[josephscott]

Both libxml2 and PHP have been updated to fix this issue. You'll need libxml2 2.7.3 and PHP 5.2.9 or higher. There's currently no official PHP release that includes this fix, so for now you'll have to use a nightly build of PHP and libxml2 2.7.3 to get the fix.

Little more info at ​http://josephscott.org/archives/2009/02/update-on-libxml2-issues/

[josephscott]

I'm not thrilled that it's come to this, but there are still hosts out there that refuse to update known broken versions of PHP+libxml2. So I've added a patch that has been circulating around to work around this.

[westi]

I am not thrilled by this.

I wonder if we would do better to create a plugin that people can install and point them at that while we have hosts not upgrading.

We could use the xmlrpc_methods filter to fix the content of $HTTP_RAW_POST_DATA (while returning whatever was passed to the filter) that way the plugin would fix it for all WordPress versions and could be installed on existing 2.7/2.7.1 installs.

What do you think?

[josephscott]

That is perfect! I've already written and tested the plugin to confirm that it works on broken install of libxml2 2.7.x:

​http://josephscott.org/code/wordpress/plugin-libxml2-fix/

I've submitted it to the plugin directory and once it goes live there I'll include a link to it at the above page and on this ticket, along with another blog post on the subject.

Once that plugin hits the plugin directory and everything looks good I'm going to close this ticket.

[josephscott]

The plugin is now available at ​http://wordpress.org/extend/plugins/libxml2-fix/

I'm closing this ticket as wontfix and pointing everyone who isn't able to upgrade PHP and libxml2 to this plugin.