REST API: Introduce a controller for searching across post types

[iseulde]

See #38920 for a simple core use case.

There seems to be no way to query posts across multiple (or all) post types as you can do with WP_Query. You have to start making multiple requests, which is an inconvenient and non-optimal approach? Ideally this should be directly passed to the server and WP_Query.

Related: #39851.

[swissspidy]

There are separate controllers for every post type, which definitely has its benefits. For queries across multiple post types, I would probably use batch requests. See ​https://developer.wordpress.org/rest-api/requests/#internal-requests.

For #38920 we could create a new endpoint that only returns the needed data without additional overhead.

[iseulde]

For that ticket it would be enough to create a separate endpoint with the needed data. Not sure if it's the best way to go in general though.

[jnylen0]

Let's look into this for 4.9.

I'm thinking that /wp/v2/posts should support a special argument like ?any_type=true or perhaps ?type=post,page, ?type=*. There may be some difficulties due to differences in response structure (in core, posts and pages are somewhat different, and attachments are very different).

[danielbachhuber]

Replying to jnylen0:

Let's look into this for 4.9.

I'm thinking that /wp/v2/posts should support a special argument like ?any_type=true or perhaps ?type=post,page, ?type=*. There may be some difficulties due to differences in response structure (in core, posts and pages are somewhat different, and attachments are very different).

For the record, this violates the principles of REST and I generally think it's a really bad idea.

Creating some batch GET endpoint would be another approach: ​https://github.com/WP-API/WP-API/issues/1506 GraphQL would be the inevitable conclusion.

[jnylen0]

Replying to danielbachhuber:

For the record, this violates the principles of REST and I generally think it's a really bad idea.

While it's true that /wp/v2/posts currently means "posts with post_type of post", at least a post of another type is still a post.

Creating some batch GET endpoint would be another approach: ​https://github.com/WP-API/WP-API/issues/1506 GraphQL would be the inevitable conclusion.

This is a far larger task, and our WP.com batch endpoint has caused us a lot of very strange issues over the years. Some of them are specific to the setup there; some would eventually reoccur in multisite context with site and theme switching. I think it would ultimately be a pretty bad idea to ship this in core.

How about a separate endpoint like /wp/v2/posts/all_types?

[danielbachhuber]

Replying to jnylen0:

While it's true that /wp/v2/posts currently means "posts with post_type of post", at least a post of another type is still a post.

This is an incorrect assumption that will likely lead to negative downstream implications. In the context of the database schema, yes different content types are stored in the same table (and history shows some of the ramifications of that).

In the context of the REST API data model, not all posts are alike — in fact, they can be more different than similar.

Is a nav_item a type of post? Is a WooCommerce product a type of post? Both are shaped drastically different than a standard WordPress post. Any client code trying to handle arbitrarily shaped data is going to be a mess.

Furthermore, I believe trying to fit a type argument into WP_Posts_Controller would be a pretty substantial monkey-patch development job. Currently, a request to /wp/v2/posts follows a different code path than a request to /wp/v2/pages. You'd need to hack around the PHP architecture to get the controllers to behave consistently.

And so on.

How about a separate endpoint like /wp/v2/posts/all_types?

This seems like a hack to me. If there's one specific feature of Gutenberg for this use-case, then I'd suggest Gutenberg handle this entirely and leave the API as it is.

[jnylen0]

This is for the feature of the link dialog that allows linking to a post or page. See #38920 for the core ticket to update this to use the REST API.

For reference, here is the code that currently drives this Ajax action: https://core.trac.wordpress.org/browser/tags/4.8/src/wp-includes/class-wp-editor.php#L1558

Thanks for explaining your concerns; I agree this isn't the best fit for the existing wp/v2 endpoints. However, shifting the responsibility to the client to batch requests isn't a great solution either, and this feature request has come up several times in other contexts, though I don't have a link at the moment.

We could write a minimal endpoint to provide this functionality and ship it in Gutenberg first.

[danielbachhuber]

For the purposes of the link dialog, I think you could make two separate requests (Posts and Pages), combine the results, and call that good enough for now. Given how much backwards compatibility Gutenberg is creating, I don't see it as entirely necessary to spend much time solving the problem for this particular use case.

[iseulde]

Starting to think it might be worth to create a separate search endpoint for things like this, as suggested earlier.

For #38920 we could create a new endpoint that only returns the needed data without additional overhead.

It would only return the common properties such as link, title, date and maybe a summary.

[iseulde]

(Because I think in the case of linking, you actually do want to search all non internal post types, such as WooCommerce products etc.)

[joehoyle]

I think per @iseulde's point, a search endpoint would be best here. As @danielbachhuber said, having /wp/posts or similar return different types seems like a really bad idea, it totally breaks out controller - per - type model, schema for endpoint etc.

I'm still not sure why a /batch wouldn't suffice here though?

[rmccue]

I am much more in favour of a search endpoint which could search across different types of entities, and I think that's probably a good endpoint to have for both backend (link dialogue, etc) and frontend (site search, autocomplete, etc) usage.

Potentially, we could add an additional context (say, context=internal) to a search endpoint to allow searching types which don't have show_in_rest, but we would want to very carefully consider how this works and the use cases for it. I think a better idea is simply to require show_in_rest.

A generic /batch could also work, but I think a targeted search endpoint is a better fit, and it also avoids the problem of boiling the ocean (by trying to solve one use case with a very generic fix). This also fits nicer with things like pagination, sorting, etc.

[kadamwhite]

I think we're out of time for 4.9. I wouldn't want to rush a change like this in at the last minute; we'll try to get some movement on this early in the next cycle. Provisionally punting. @rmccue and @jnylen0 feel free to override if you disagree.

[afercia]

Discussed this issue during today's Gutenberg bug-scrub and heard there are dissenting opinions in the REST API team. I'd propose to do a quick use case example:

Say I’m writing a post and I want to promote one of my products. I want to link to my product. Are you really saying in Gutenberg I have to do that manually? Do you realize this won't be very... welcomed by users?

As I see it, and I'm pretty sure I'm not the only one, having the ability to link to any post type is paramount in a modern editor. Without this ability, Gutenberg will be pretty limited. At that point, better to switch back to AJAX :D

[joehoyle]

Say I’m writing a post and I want to promote one of my products. I want to link to my product. Are you really saying in Gutenberg I have to do that manually? Do you realize this won't be very... welcomed by users?

This is still possible in the REST API, just with multiple requests. It's up to the client on how it wants to present multiple types to the user. There's the api client (in this case gutenberg) that can do things for the user, like send multiple requests. Different post types can be very different, so the client should handle these as such. For example, don't assume you can just get "all posts across all types" and then display them. What if a post type doesn't support titles, etc. Different post types can be as different as a Tag and a Post, as far as the API is concerned. That's why the suggestion is a search endpoint (or similar) that would return multiple groupings of different objects.

[danielbachhuber]

Looping back on this issue, I think we need a read-only wp/v2/search endpoint that returns matching objects with title, link, and possibly excerpt.

By default, the endpoint would only return searchable posts where show_in_rest=true. However, a plugin could easily register more fields to the returned objects, use ElasticSearch to pull in a mixture of data (comments, users, etc.), and so on.

[iseulde]

@danielbachhuber Yes, I think that's exactly what we need. :) Maybe also with a date. I like that this could be extended beyond posts to include e.g. category or author archives.

[flixos90]

An initial implementation is available at ​https://github.com/WordPress/gutenberg/pull/6489. I'll write tests for it later this week.

[danielbachhuber]

@flixos90 Do you want to prepare a patch to bring this over into core?

[flixos90]

@danielbachhuber Sounds good! I should be able to get to it next week.

[danielbachhuber]

39965.1.diff ports over an ​adaptation of the search controller in Gutenberg:

• Introduces a WP_REST_Search_Controller class which registers a /wp/v2/search endpoint.
• Search types are handled by extending WP_REST_Search_Handler. The default search type is WP_REST_Post_Search_Handler but can be filtered by plugins or a theme.
• Includes relevant unit tests.
• Updates all @since references to 5.0.0 and removes the 'gutenberg' textdomain.

[danielbachhuber]

39965.2.diff changes the use of 'dummy' to 'test'.

[danielbachhuber]

In 43739:

REST API: Introduce controller for searching across post types.

Introduces a WP_REST_Search_Controller class which registers a /wp/v2/search endpoint. Search types are handled by extending WP_REST_Search_Handler. The default search type is WP_REST_Post_Search_Handler but can be filtered by plugins or a theme.

Props danielbachhuber, flixos90, pento, rmccue.
Fixes #39965.

[danielbachhuber]

Reopening for merge to trunk.

[kadamwhite]

@danielbachhuber: @joehoyle noticed that we made a mistake on one of our version strings, see https://core.trac.wordpress.org/browser/branches/5.0/src/wp-includes/rest-api/endpoints/class-wp-rest-search-controller.php?rev=43739#L73

_doing_it_wrong( __METHOD__, sprintf( __( 'REST search handlers must extend the %s class.' ), 'WP_REST_Search_Handler' ), '5.5.0' );

5.5.0 is getting a bit ahead of ourselves :)

[danielbachhuber]

In 43741:

REST API: Fix version number in _doing_it_wrong() call.

_doing_it_wrong(), indeed.

Props joehoyle.
See #39965.

[pento]

In 44107:

REST API: Introduce controller for searching across post types.

Introduces a WP_REST_Search_Controller class which registers a /wp/v2/search endpoint. Search types are handled by extending WP_REST_Search_Handler. The default search type is WP_REST_Post_Search_Handler but can be filtered by plugins or a theme.

Merges [43739,43741] from the 5.0 branch to trunk.

Props danielbachhuber, flixos90, pento, rmccue, joehoyle.
Fixes #39965.