Large file upload through XML-RPC fails with error 500

[avibrazil]

It fails through XML-RPC but succeeds through the web UI.
File size is about 40MB and PHP upload limits are correct and higher than that.
XML-RPC upload fails either through HTTP and HTTPS, either through IPv4 or IPv6.
Using netstat in another terminal, I can see large amounts of bytes being transferred for a while.
But it suddenly fails with error 500:

xmlrpc.client.ProtocolError: <ProtocolError for abstra.to/xmlrpc.php: 500 Internal Server Error>

On the server side there is absolutely nothing on Apache error log, not a single char being logged, and just this on access log:

2804:14c:183:83ce:631c:9fb1:3797:93a7 - - [14/Apr/2017:03:43:15 -0700] "POST /xmlrpc.php HTTP/1.1" 500 3375 "-" "Python-xmlrpc/3.5"

Log says 3375 bytes but netstat shows a higher number of bytes being transferred for several minutes.
XML-RPC upload of smaller files works — successfully tested with 12MB.

I would investigate this as a typical network error but doesn't look like, since upload never fails when making this same transfer through the web UI, from same Linux client machine. I would also discard lack of RAM and lack of storage on the server because same file upload through web UI works.

The server apparently receives the whole blob (it stays connected for several minutes) but apparently fails in some internal file operation after it finishes receiving it all.

This wordpress site will be a podcast server and I'm automating publication and media uploading with a script that uses ​https://python-wordpress-xmlrpc.readthedocs.io/en/latest/

Thank you in advance

[redsweater]

Based on my experience working with the limitations of the WordPress XMLRPC API, I would say this issue is almost certainly based on the size of the image somehow overwhelming the server's PHP instance such that it can't handle the upload.

To upload files via XMLRPC the client has to convert the image to base64 and encode the entire file into a single POST request to the XMLRPC API endpoint. At this point the server process, in PHP, has to unwrap the entire XML file and handle each of the arguments separately. In the case of file data this means decoding from base64 back to binary data, and then performing whatever additional processing is necessary.

There may be tweaks that could be done in the XMLRPC library or in WordPress to treat file uploads differently from other API requests, but this is a long-lived problem that seems unlikely to be worth addressing if the newer REST API handles it more efficiently. (​https://developer.wordpress.org/rest-api/reference/media/#create-media)

Uploads via the new REST API are handled more similarly to the way uploads are handled by the web-based uploads. That is: the binary data for the upload is POST'd directly as the content of a request to upload data. This should give the API the ability to more efficiently process the data and stream it for example to disk without imposing the same memory requirements on the PHP process. I suspect the new REST API is far more resilient to large file uploads than XMLRPC is.

[redsweater]

It's worth noting that you should be able to mix and match with the old XMLRPC API and the newer REST API. If you've already invested heavily in an XMLRPC solution you could consider using the REST API just for uploading files, and keep using XMLRPC API for everything else.

[avibrazil]

This is cool, I was not aware of this REST API. Is it available right now on stable WP releases or it is a plugin I'll have to install ?

Also, is there any Python lib that puts some semantic on top of the WP REST API to make things easier and code cleaner ?

By the way, I'm uploading 50MB media files (MP4), not images. But this detail makes little difference for my problem.

Thank you in advance.

[redsweater]

Hi - to clarify, I am not an active member of the WordPress team but just chiming in because I have some knowledge in this area. The WordPress REST API is indeed part of the standard open source releases since a relatively recent update. It's also available on WordPress.com.

I don't know anything about Python libs on top of the API. It's pretty new so I woudn't be surprised if there aren't many wrappers yet.

I noticed about the audio files aspect of your post and changed most references to images, in my response, to files, but I missed a couple :)

[rmccue]

The REST API shipped in WordPress 4.7 as part of core, however we don't currently have any external authentication shipped with core (alas). That's an ongoing project, but isn't scheduled for a release yet, so there's no set timeline on that shipping. (There are plugins available for this, including our official ​OAuth 1.0a plugin, or ​Basic Auth which works the same as XML-RPC's authentication.)

REST API uploads can work in one of two ways: 1) the same as browser-based (content type set to multipart/form-data, data encoded like a browser upload), or 2) direct upload (content-type set to your actual type, like image/png, and data directly in the request body). Both are subject to PHP's regular data input limits.

We're exploring adding support for chunked uploads in #40410, which would allow you to upload in (e.g.) 5MB chunks then reassemble into a single file.

As for the error here, WordPress does do some processing on uploads, so potentially it's running out of memory while it's trying to process this. XML-RPC should do the same processing, except that internally it first needs to decode the XML request, and I suspect that this is where the issue is coming from. (If so, this would be fixed by using the REST API.) Your PHP error logs (not your server access logs) should reveal more about that. However, that's more of a support thing, which is likely better posted at ​https://wordpress.org/support/

I suspect this isn't a bug, but rather just a limit of the server that you're hitting, so marking this as a candidate for close pending your response.

[redsweater]

Thanks @rmccue for the detailed response! And I didn't realize the REST API is not yet suitable for authenticated access. Good to know!

[avibrazil]

Replying to rmccue:

Your PHP error logs (not your server access logs) should reveal more about that.

The PHP errors are directed to web server's error log. As I said before, not a single char is being logged here. This alone is a good enough reason to investigate, even if REST is now the new way to go.

[dd32]

Replying to avibrazil:

Replying to rmccue:

Your PHP error logs (not your server access logs) should reveal more about that.

The PHP errors are directed to web server's error log. As I said before, not a single char is being logged here. This alone is a good enough reason to investigate, even if REST is now the new way to go.

That's dependant upon the server configuration, PHP may even be set not to log anything.
For us to proceed any further with this report it'd be great to get some information from you which directs us to exactly what is causing the 500 - we could probably reproduce, but it'd just be an assumption that we're hitting the same issue you are.

You may find that adding define( 'WP_DEBUG', true ); and/or define( 'WP_DEBUG_LOG', true ); to your wp-config.php file will be needed (depending on server configuration). Doing the latter should cause the PHP logs to be logged into wp-content/debug.log instead of vanishing as in your case.

[hellofromTonya]

As this ticket is marked for close and it's been 4 years since feedback was requested, I'll go ahead and close this ticket.

For us to proceed any further with this report it'd be great to get some information from you which directs us to exactly what is causing the 500 - we could probably reproduce, but it'd just be an assumption that we're hitting the same issue you are.

However, if this is still an issue, @avibrazil please provide feedback for further investigation.