Proposal: Standardize WP Responses to JS Requests as JSON

[filosofo]

Summary

I propose that we standardize Ajax actions on the ​JSON RPC 2.0 spec and provide a JSON-RPC API for querying content.

Motivation

Inconsistent Admin Behavior

Currently, when JS in a user's client (browser) makes a request to WP, the response can appear in a variety of forms: integers, JSON, XML, or sometimes nothing at all.

Because WP has such a variety of responses, that means in similar situations it's using different ways to make requests and handle responses. Doing similar things a bunch of different ways increases the maintenance overhead and therefore the likelihood of there being bugs. It also means that future implementations and plugin authors have little guidance on best practice.

JS Query API

There seems to be interest both in querying comments and ​posts in a data exchange format easily adapted by JavaScript.

Proposal: Core JSON RPC

Why any standardization?

• A single way of doing things means for programmers that you don't repeat yourself (DRY)
• A standard way of handling such requests focuses attention on a few sections of code, so bugs are noticed sooner and fixed in a wider range
• Standardization gives guidence to future development, both of core and plugins
• Standardization provides greater possibility of interactivity among core, plugins, and other sites and applications.

Why JSON?

In transmitting data JSON has at least several advantages:

• It's lightweight in content, using less bandwidth than for example XML.
• It's well-supported across a number of languages, most notably JavaScript (and PHP 5).
• It seems to be growing in popularity

Why a remote procedure call (RPC)?

• Most of the Ajax in WP is in the form of an RPC.
• We could appropriate some of the code used already with XML-RPC in WP.
• It's a common way of doing something we need: standardized ways to initiate actions on a remote computer

Why JSON RPC in particular?

• JSON RPC allows requests to specify an ID which the response also includes. The ID allows a client to make asynchronous requests yet avoid race conditions, by matching request to response.
• JSON RPC uses some of the same error codes as XML-RPC, and using standardized error codes creates consistency and easier localization.

Admin Ajax

Admin ajax requests and responses should be converted to use the JSON 2.0 RPC.

For example, a request to delete a comment could look like the following:

{"jsonrpc": "2.0", "method": "wp.deleteComment", "params": [123], "id": 4567}

Note that the wp.deleteComment method is already defined for XML-RPC; using a common interface would give greater attention to XML-RPC and would improve consistency across the board.

A response to the previous might look like this:

{"jsonrpc": "2.0", "result": {"action":"deleted", "commentID":123}, "id": 4567}

WordPress would provide the JS for making the JSON requests, attaching listeners and matching responses to the appropriate callbacks by ID. Perhaps like so:

wpMakeJSONQuery('wp.deleteComment', [123], myCallback, myErrorCallback);

function myCallback( result ) {
   if ( 'deleted' == result.action )
	alert('Comment ID #' + result.commentID + ' deleted!');
}

function myErrorCallback( code, message, data ) {
	data = data || {};
	alert('Error: ' + message );
}

Your thoughts?

[scribu]

I completely agree that all AJAX requests should return JSON.

The JSON RPC idea makes sense too.

However, what would happen to all the plugins that register their own AJAX requests?

Moving out of 3.1 milestone until we can commit to it.

[filosofo]

Replying to scribu:

However, what would happen to all the plugins that register their own AJAX requests?

We could continue to use, but deprecate, the old API. The JSON-style requests are easy enough to distinguish from the old-style ones, and so the responses can be easily differentiated.

The reason I chose milestone 3.1 is that with all the Ajax stuff coming in from your GSoC project (and others?) it's a good time to standardize.

[dd32]

This gets a huge +1 from me, I know how difficult it is to follow some of the wpAjax code, a near impossibility for many new developers..

[scribu]

[15604] would be a good example of why we need this.

[frsh]

This is a fantastic idea. I would love to utilize JSON in the iOS app.

[Viper007Bond]

I would like to subscribe to your newsletter.

[josephscott]

I followed the JSON-RPC development for awhile, one thing they never addressed very well (to my knowledge) was binary data. This is also the single biggest weakness in XML-RPC from my point of view. For *RPC style APIs binary data ends up being base64 encoded, which is an added burden that isn't really needed. Being able to do a direct HTTP POST with raw binary data is both easier and more efficient.

Another, often less observed, weakness is no specific way to version the API. Contrast this with an HTTP based API, which can include version numbers as part of the URL string (just as an example: /json-api.php/wp-delete-comment/1.0/).

I've yet to find a compelling reason to use an RPC style API instead of a general HTTP one. I'm happy to be convinced other wise.

As for the general idea, this strikes me as something that would do well as a core plugin, something that ships with WP and is even enabled by default, but is developed and updated as a plugin. This allows for advancements and bug fixes that aren't limited by the overall WP release schedule.

Something you may want to take a look at as well is ​http://wordpress.org/extend/plugins/json-api/ - sadly I haven't spent time playing with this yet, but I think it is worth reviewing in conjunction with this ticket.

[filosofo]

Replying to josephscott:

I followed the JSON-RPC development for awhile, one thing they never addressed very well (to my knowledge) was binary data. This is also the single biggest weakness in XML-RPC from my point of view. For *RPC style APIs binary data ends up being base64 encoded, which is an added burden that isn't really needed. Being able to do a direct HTTP POST with raw binary data is both easier and more efficient.

What is the current use case in which you're posting binary data via XHR?

Another, often less observed, weakness is no specific way to version the API. Contrast this with an HTTP based API, which can include version numbers as part of the URL string (just as an example: /json-api.php/wp-delete-comment/1.0/).

You could easily add an argument to the endpoint to indicate version:

admin-ajax.php?json-api=1.0

or pass it as one of the JSON parameters ({json-api-version: "1.0"}).

I've yet to find a compelling reason to use an RPC style API instead of a general HTTP one. I'm happy to be convinced other wise.

I listed some in the OP. I've been using my JSON RPC plugin on a number of sites now, and the biggest benefits are:

• Standardization: I don't have to worry about any other Ajax functionality or libraries: I just check for the JS methods and call them if available.
• Consistent Response Handling: Responses can be only one of the following:
  • Syntactically invalid response (this includes blank responses)
  • Successful response
  • Error response

All are easy to check for in the JSON RPC API, making it easy to handle problems. In contrast, the current core Ajax responses can take any of a number of forms, including no response for a successful request. This makes it difficult to test and to use by plugins. It also requires that each piece of code making a request include the logic necessary to determine success or failure, which is needless redundancy.

As for the general idea, this strikes me as something that would do well as a core plugin, something that ships with WP and is even enabled by default, but is developed and updated as a plugin. This allows for advancements and bug fixes that aren't limited by the overall WP release schedule.

Perhaps. Ultimately it's not much use if plugin and theme developers can't use it.

I'll post a link to my plugin below, although I still want to verify that it works with 3.1 and make a sample use plugin, before doing a real release.

[filosofo]

Here's my ​JSON RPC API plugin.

This is rough; feedback is welcome.

[josephscott]

Replying to filosofo:

What is the current use case in which you're posting binary data via XHR?

Uploading images (or any other media file) is the obvious target. Also, any generalized API like the one proposed will not be limited to just XHR use.

You could easily add an argument to the endpoint to indicate version:

admin-ajax.php?json-api=1.0

or pass it as one of the JSON parameters ({json-api-version: "1.0"}).

True, I should have been more specific, I was thinking of per method versions.

• Standardization: I don't have to worry about any other Ajax functionality or libraries: I just check for the JS methods and call them if available.
• Consistent Response Handling: Responses can be only one of the following:
  • Syntactically invalid response (this includes blank responses)
  • Successful response
  • Error response

I don't see that any of these reasons require an RPC approach.

[filosofo]

Replying to josephscott:

Replying to filosofo:

What is the current use case in which you're posting binary data via XHR?

Uploading images (or any other media file) is the obvious target.

True XHR uploads are very fringe. They're not going to be supported across enough browsers any time in the near future and aren't being used in current WP.

I'm not saying everything asynchronous should use also JSON RPC; just most of the stuff that's communicating information, executing commands, etc.---the vast majority if not everything currently being done in WP.

I'm trying to figure out why you think support for something we're not doing currently and likely won't be able to do for years is particularly relevant. But sure, we could allow exceptions for binary data Ajax.

Also, any generalized API like the one proposed will not be limited to just XHR use.

True. But whatever is going to make use of the API in this future state will have more choices under my proposal; it won't be harmed by it.

True, I should have been more specific, I was thinking of per method versions.

Which can be solved the same way. That seems to be how many popular web APIs handle such things; take a look for example at the Google Maps and MailChimp API versioning.

I don't see that any of these reasons require an RPC approach.

You're right that they don't require this specific approach. However, they do require some sort of standardization as opposed to the current eclecticism. JSON RPC has the advantage of being an already-existing standard, widening support and familiarity throughout the web dev world, being easy to use in JS, among other things.

However, I'm reasonably certain another communication standard could work almost as well. My point is pushing to a decent standard, not finding the absolutely perfect one. Do you have an alternative in mind?

[dd32]

It should be worth noting here, That we do actually have a "standard" library for communication, the WP_Ajax_Response PHP Class and the WP Ajax Javascript library..

JS Library: ​http://core.trac.wordpress.org/browser/trunk/wp-includes/js/wp-ajax-response.dev.js An implementation of it as a reciever: ​http://core.trac.wordpress.org/browser/trunk/wp-includes/js/autosave.dev.js#L85

The PHP Class: ​http://core.trac.wordpress.org/browser/trunk/wp-includes/classes.php#L1673 An implementation of sending it: ​http://core.trac.wordpress.org/browser/trunk/wp-admin/admin-ajax.php?rev=15956#L363

The major problem with these libraries, is that people do not understand how to use them.. The 0/1 responses are valid and carry a known explanation, and the XML structures returned are also of a standard quantity.

If anything, This ticket should focus on integrating well with jQuery as well as being simple/powerful enough for 99% of developers to understand the code.

[filosofo]

Replying to dd32:

It should be worth noting here, That we do actually have a "standard" library for communication, the WP_Ajax_Response PHP Class and the WP Ajax Javascript library..

In a very limited sense, it can handle integer responses. It badly mixes presentation with data (note how you're expected to pass a DOM element for it to put response messages inside of). It doesn't do much to help you with non-integer responses or syntactically invalid responses. The other option is a DOM fragment with certain child elements and attributes.

More importantly, it provides no standard for handling remote procedure calls: you can throw instructions at the server in any old way, and that's basically what happens throughout core and plugins.

The major problem with these libraries, is that people do not understand how to use them.. The 0/1 responses are valid and carry a known explanation, and the XML structures returned are also of a standard quantity.

The major problem with these libraries is that they assume a narrow range of uses. XML's fine, but it does use more bandwidth for the same information than JSON.

If anything, This ticket should focus on integrating well with jQuery as well as being simple/powerful enough for 99% of developers to understand the code.

jQuery is neither here nor there with respect to JSON, as it doesn't provide a way to encode JS objects into JSON. If you have to rely on native methods or a JSON library anyways (my plugin's using Crockford's), there's no point in slapping on a 77k minified library just for 20 lines of XHR code. Note that my plugin's minified JS is 5k for everything: the JSON multi-browser compatibility, XHR, and JSON RPC-specific functionality.

The standard of method, data, and response handlers makes it very easy for developers to understand, much more so than an anything-goes structure.

[josephscott]

Replying to filosofo:

True XHR uploads are very fringe. They're not going to be supported across enough browsers any time in the near future and aren't being used in current WP.

Also, any generalized API like the one proposed will not be limited to just XHR use.

True. But whatever is going to make use of the API in this future state will have more choices under my proposal; it won't be harmed by it.

You asked for an example, I provided one. JSON-RPC will suffer from the same problem XML-RPC does in this area.

True, I should have been more specific, I was thinking of per method versions.

Which can be solved the same way. That seems to be how many popular web APIs handle such things; take a look for example at the Google Maps and MailChimp API versioning.

I scanned through the docs for both of these, I didn't see JSON-RPC listed as an option for either one. Quite the opposite, they use more standard direct HTTP APIs, as I suggested in my first comment.

I don't see that any of these reasons require an RPC approach.

You're right that they don't require this specific approach. However, they do require some sort of standardization as opposed to the current eclecticism. JSON RPC has the advantage of being an already-existing standard, widening support and familiarity throughout the web dev world, being easy to use in JS, among other things.

Perhaps I stopped following the development of JSON-RPC too soon then. I hadn't seen much uptake in JSON-RPC, though I might not be looking in the right places.

However, I'm reasonably certain another communication standard could work almost as well. My point is pushing to a decent standard, not finding the absolutely perfect one. Do you have an alternative in mind?

As I suggested in my first comment on this ticket, my suggest is to use a simple direct HTTP based API. Not an RPC based API.

The Twitter and Facebook Graph APIs are popular examples of what I'm suggesting.

[filosofo]

Replying to josephscott:

You asked for an example, I provided one. JSON-RPC will suffer from the same problem XML-RPC does in this area.

Joseph, you said that the lack of binary support was "the single biggest weakness in XML-RPC from my point of view." For it to be a weakness relevant to this proposal (as opposed to XML/JSON RPC in the abstract), it would need to have some bearing on likely use.

Which can be solved the same way. That seems to be how many popular web APIs handle such things; take a look for example at the Google Maps and MailChimp API versioning.

I scanned through the docs for both of these, I didn't see JSON-RPC listed as an option for either one. Quite the opposite, they use more standard direct HTTP APIs, as I suggested in my first comment.

Sorry not to be clear. I meant using the endpoint to indicate API version rather than each method, because you said "I was thinking of per method versions."

Perhaps I stopped following the development of JSON-RPC too soon then. I hadn't seen much uptake in JSON-RPC, though I might not be looking in the right places.

I meant JSON in general.

As I suggested in my first comment on this ticket, my suggest is to use a simple direct HTTP based API. Not an RPC based API.

The Twitter and Facebook Graph APIs are popular examples of what I'm suggesting.

I'm not sure what you're arguing against. Are you arguing against using JSON as a format for request and response? Then my response is:

• JSON is used in response by both examples you cite, Twitter and Facebook.
• It seems to be growing in popularity. I admit I can't provide numbers to demonstrate that conclusively.
• It's lightweight.
• It works well with a lot of languages; JS most obviously and relevantly.

Are you arguing against having any protocol for remote procedure calls? Then my response is:

• Standardization is good for the reasons I've mentioned several times so far (and without apparent disagreement).
• Putting the RPC methods in an easy-to-parse request object avoids potential namespace collisions and the ambiguities of parsing a request URL.
• Having a third-party protocol is generally better than coming up with your own, both in terms of work for yourself and others, in maintenance and learning the syntax.
• We already have a ton of WP methods defined for XML RPC, which we can re-use for JSON RPC.

[josephscott]

I have no problem with JSON at all, it is my preferred serialization format. I mentioned in my first comment my concern is with using an *RPC approach to a new API like this. A simple HTTP (REST, though that term is generally mis-understood and abused) would be my preferred approach. That is why I asked for advantages that *RPC provided that you couldn't do with simple HTTP and gave a specific weakness based on my experiences with XML-RPC. Ignoring the obvious weakness because you haven't run into yet doesn't make it go away.

I think there is definitely positive aspects to *RPC, but I consider base64 encoding to be a near fatal blow. This is one point that AtomPub has a clear advantage over XML-RPC.

[scribu]

A good read on XML vs. JSON:

​http://norman.walsh.name/2010/11/17/deprecatingXML

[filosofo]

Another thought about JSON-RPC: probably the best candidate to replace the current uploader (in the variety of ways it does uploads and because it's from the TinyMCE folks) is Plupload, and it handles responses to upload events with JSON-RPC.

[hakre]

Related: #11175, #15827

[MikeSchinkel]

Replying to josephscott:

A simple HTTP (REST, though that term is generally mis-understood and abused) would be my preferred approach.

+1 that a general API for WordPress sites would be RESTful and not RPC. RPCs are becoming the anti-pattern for web APIs where REST APIs (mis-understand and abuse) are becoming the standard. You'll find very few public web APIs offered as JSON-RPC or even XML-RPC vs. REST (24+169 vs. 5348):

• ​http://www.programmableweb.com/apis/directory/1?protocol=JSON-RPC (Count: 24)
• ​http://www.programmableweb.com/apis/directory/1?protocol=XML-RPC (Count: 169)
• ​http://www.programmableweb.com/apis/directory/1?protocol=REST (Count: 5248)

Here's a few good articles on the topic:

• ​http://www.emergentone.com/blog/api-types-rest-vs-rpc/
• ​http://blog.perfectapi.com/2012/opinionated-rpc-apis-vs-restful-apis/

One of the biggest benefit of REST vs. RPC is REST uses the natural caching mechanisms of HTTP.

That said, I do agree adding JSON-RPC support would be great in the short term, see: #23099.

[rmccue]

For those interested, I've been working on the initial version of a JSON API over on ​a Gist with feedback from a few people on Twitter and IRC.

I'd like to recommend closing this topic though, since we have #23099 for JSON-RPC support (despite it being newer, it's more focussed and has patches) and opening a new discussion for a REST JSON API on make/core (nacin said he'd get to this post-3.5.1).

[scribu]

Duplicate of #23099.