REST API: extend _fields parameter to selectively include nested fields in response JSON

[kadamwhite]

This enhancement has been broken out from #38131, which introduced the _fields query parameter to whitelist select fields for inclusion in the API response object.

In the comments of that ticket @TimothyBlynJacobs uploaded a patch with support for nested fields, such as returning title.raw while omitting title.rendered:

[patch 31131.3.diff allows] specifying title.raw for just the raw title or title to retrieve both the raw and rendered title. This could also be used by controllers to only calculate the fields as necessary. I'm not sure whether this should be used to limit the fields queried ( I think that can be done in WP_User_Query but I'm not sure about the others ). I imagine some of the performance benefits here, besides response size, would be not filtering the_content if it wasn't a requested field, for instance.

There's been discussion of how feasible it is to use these parameters to hint that certain types of data processing are unnecessary, but on its own the base use-case of "give me the raw API response without its rendered counterpart" in edit mode could significantly speed up initial load of editor interfaces on slow connections.

This nesting feature has also been proven out in the plugin repository in ​the rest-api-filter-fields and ​wp-rest-mespath, the former through the same basefield.nestedfield syntax and the latter through JMESPath.

[kadamwhite]

Basic unit test exploring the proposed dot-nested ?_fields syntax

[kadamwhite]

Added a patch with a basic unit test exloring the dot-nested syntax proposed above. I propose that the dot-separated solution is preferable to JMESPath for core usage, as it will be a straightforward enhancement over the initial syntax from #38131.

[kadamwhite]

Updates from REST API chat discussion: This has renewed relevance given ​discussions within the Gutenberg repo around how to signal that we can skip the expensive generation of content.rendered when the filtered value is not used. Implementing that in Gutenberg is non-trivial, but this _fields improvement is a good first step and would yield immediate performance gains for client-side consumers even without any special handling within our controllers.

In Slack @danielbachhuber proposes ?_fields=content[raw] as an alternative to either of the syntaxes discussed above.

[danielbachhuber]

I'm amenable to dot syntax too (i.e. ?_fields=content.raw). My original suggestion for ?_fields=content[raw] was to follow PHP's existing pattern for transforming ?bar[]=1&bar[]=2&bar[]=3 into $_GET['bar'] = array(1, 2, 3).

We could support both if it's not too difficult. Otherwise, I'm fine with dot syntax if that's what everyone else prefers.

One minor backwards compatibility concern: a field could theoretically be named 'content.raw'. In this scenario, ?_fields=content.raw should return the 'content.raw' field, not a 'content' field with sub-field 'raw'.

[desrosj]

This still needs some work. Going to punt it. @kadamwhite if you have the time the next two days before 5.2 beta 1 and feel confident in this, feel free to add it back.

[kadamwhite]

Propose an implementation for nested _fields handling

[kadamwhite]

I have uploaded a patch with a proposed solution to this problem. It works in three parts:

• The initial generation of the accepted fields list is modified to be nested.field-aware,
• The posts controller is modified to use a smarter helper method to determine if a specific property should be included, rather than a simple in_array call, and
• The filter which strips invalid properties out of the response is modified to do a recursive check against a tree generated from the _fields input string. (Thank you @rmccue for helping with the code here.)

I would love to get some critique on this patch.

My biggest dissatisfaction with it is the code organization: the recursive key merge method feels like the simplest solution to this problem, but making that a rest_-prefixed global function feels inappropriate. However, introducing an arbitrary wp_ helper method also feels like an overstep. I'd welcome input, @danielbachhuber @desrosj @timothybjacobs

One additional unit test I'd like to add would be a check to ensure the content filters are not triggered if content.rendered is not requested in the fields list.

Once we agree on the mechanics of the patch, I can update the remaining controllers to use this new method when determining what fields to compute.

[TimothyBlynJacobs]

FWIW, I think extracting those as generic array utilities would be immensely helpful. I've had to implement similar things in other contexts, and having core functions for that would be helpful.

Perhaps the filtering by dotted fields could be a part of WP_List_Util? And then extract is_field_included into a standalone function?

[rmccue]

Replying to TimothyBlynJacobs:

FWIW, I think extracting those as generic array utilities would be immensely helpful. I've had to implement similar things in other contexts, and having core functions for that would be helpful.

Definite +1 on extracting this out. Having an ​idx-style function in core would be great, along with other utillity functions.

[kadamwhite]

I think an idx-style method is out of scope here, but I've refreshed the patch and extracted the utility into rest_is_field_included. I think it makes more sense here than in a list utilities context because it's both not strictly working from a list, and uses hierarchy logic that's specific to the REST API.

[dlh]

I haven't been able to look at this patch in too much depth, but one small thing I noticed is that I think the $permalink_template_requested and $generated_slug_requested variables could be set using rest_is_field_included(), rather than calling if ( $var && rest_is_field_included() ). Both of those variables are currently set using the in_array() checks that are replaced elsewhere in the patch with rest_is_field_included().

[kadamwhite]

@dlh good catch, thanks!

Going to commit tomorrow unless any other feedback comes in. Thanks, all.

[kadamwhite]

In 46184:

REST API: Support dot.nested hierarchical properties in _fields query parameter.

Enable clients to opt-in to receipt of one or more specific sub-properties within a response, and not other sub-properties.
Skip potentially expensive filtering and processing for post resources which were explicitly not requested.

Props kadamwhite, TimothyBlynJacobs, dlh.
Fixes #42094.

[adamsilverstein]

This is a great feature, thanks for committing this @kadamwhite! Adding needs-dev-note so we make sure developers know about this improvement.

[kadamwhite]

Dev note: ​https://make.wordpress.org/core/2019/10/10/filtering-nested-rest-response-_fields-in-wp-5-3/