Make WordPress Core

Opened 7 years ago

Closed 4 years ago

Last modified 4 years ago

#41287 closed enhancement (fixed)

Add AND query mode for taxonomy terms

Reported by: sebbb's profile sebbb Owned by: dlh's profile dlh
Milestone: 5.8 Priority: normal
Severity: normal Version: 4.8
Component: REST API Keywords: dev-feedback has-patch has-unit-tests commit needs-dev-note
Focuses: Cc:

Description

I'm trying to query the Rest API to get only posts that contain a given set of tags. If, for example, I request the following from the Rest API:

/wp-json/wp/v2/posts?tags=41+50

I get posts that don't satisfy all the tags. For example, the first post returned in the JSON response has the tags 45, 38, 46, 42, 35, 50, 43, 36, 37, 75, 76. The post doesn't have the tag with ID 41, while the expected output would be only posts that satisfy all the provided tags.

It looks like behind the scenes the API uses an OR logic instead of the correct AND logic. It returns posts that contain at least one of the provided tags. I'm experiencing the same exact problem with categories as well.

I tried different versions of the above URL, but I'm facing the same issue:

/wp-json/wp/v2/posts?tags=41&tags=50

/wp-json/wp/v2/posts?tags=41,50

Interestingly, tags_exclude seems to work perfectly and the following url correctly returns posts that have neither the tag ID 41 nor the tag ID 50:

/wp-json/wp/v2/posts?tags_exclude=41+50

Attachments (2)

41287.diff (11.2 KB) - added by earnjam 7 years ago.
41287.2.diff (6.9 KB) - added by earnjam 6 years ago.

Download all attachments as: .zip

Change History (31)

#1 @kadamwhite
7 years ago

It looks like behind the scenes the API uses an OR logic instead of the correct AND logic.

Unfortunately this is exactly what is happening; the API does not currently support AND logic for querying records by taxonomy ID. We tracked this problem in our old GitHub repository as issue # 2768, but it doesn't look like that issue has made it here into trac -- thanks for opening the ticket.

We should have a way to query posts by taxonomy terms using an AND relation. I agree that the current behavior is highly unintuitive.

@rmccue may be able to clarify but I'd also comment that, as far as I understand it, id1+id2 or id1,id2 are both just proxies for the ?tags=id1&tags=id2 syntax, so there isn't a clear way to distinguish between AND or OR without introducing a new API query parameter.

#2 follow-up: @LewisCowles
7 years ago

Is there a way for the existing OR functionality / bug to be provided via API endpoint or parameter to the existing endpoint to toggle between AND / OR?

I'm thinking it could be helpful for certain situations (find all posts in EITHER taxonomy X or Y, rather than find all posts in ONLY taxonomy X and Y).

Also does it only affect tags?

Last edited 7 years ago by LewisCowles (previous) (diff)

#3 in reply to: ↑ 2 @sebbb
7 years ago

Replying to LewisCowles:

Also does it only affect tags?

It seems to be affecting categories as well.

#4 @LewisCowles
7 years ago

Thanks. The reason I ask is that I'm going to borrow that bugged code. For a while now I've been thinking about provisioning a way for users to de-select taxonomy data to hide things they don't want. (Imagine if you could choose to hide unwanted political commentary).

It would require some work on author and editor part, but it's the first time I've seen something pre-built and will save me a few rainy days.

#5 @jnylen0
7 years ago

  • Keywords needs-patch needs-unit-tests added
  • Milestone changed from Awaiting Review to Future Release
  • Summary changed from Getting posts that belong to multiple tags or categories to Add AND query mode for taxonomy terms
  • Type changed from defect (bug) to enhancement

This is intended behavior, not a bug. I would expect "filter the posts list to posts that include tags 41,50" to return all posts that match any of these tags. "exclude tags 41,50" should not match any of those posts. If tags used AND, but exclude_tags used OR, this would be less consistent.

Having said that, adding an AND query mode would be a nice enhancement. The solution will likely look similar to the approach discussed in #39494.

For now, it should be possible to use the rest_{$this->post_type}_query filter -- possibly combined with register_rest_field (docs) -- to make this functionality work in a plugin.

Last edited 7 years ago by jnylen0 (previous) (diff)

This ticket was mentioned in Slack in #core-restapi by kadamwhite. View the logs.


7 years ago

#7 @sebbb
7 years ago

It's too bad that this is not considered a bug 😢.

I would think that the alternate syntax using + (i.e.: tags=41+50) implies that only the posts that have both tag 41 and tag 50 should be returned.

I'm working on an app that needs that feature, so I'll certainly look into a DIY solution in the meantime.

Unfortunately I don't have any PHP, WordPress or API design background myself, so I wouldn't be able to create the patch.

#8 @dd32
7 years ago

#41908 was marked as a duplicate.

@earnjam
7 years ago

#9 @earnjam
7 years ago

  • Keywords dev-feedback has-patch has-unit-tests added; needs-patch needs-unit-tests removed

In 41287.diff, I've added support for a new {$tax}_and parameter which queries using an AND relationship between terms. This follows the model of the existing parameter {$tax}_exclude for a NOT IN type query.

I also considered something like {$tax}_strict for the parameter name, but went with _and because it more closely aligns with existing WP_Query arguments for tag__and and category__and

Also, it may be beyond the scope of this ticket, in which case I can split it off into a new ticket, but I also added a new parameter, relation, which allows you to specify an OR relationship between multiple taxonomies being queried. This more closely mimics the functionality for a tax_query argument on WP_Query.

So basically this is the current behavior:

URL Tax Query Matches
?tags=1,2 tag 1 OR tag 2
?tags=1&categories=3 tag 1 AND category 3
?tags=1,2&categories=3 (tag 1 OR tag 2) AND category 3
?tags=1,2&categories=3,4 (tag 1 OR tag 2) AND (category 3 OR category 4)

With this patch, you can use a {$tax}_and type parameter to force AND matching on the terms within a specific taxonomy.

URL Tax Query Matches
?tags_and=1,2 tag 1 AND tag 2
?tags_and=1,2&categories=3 tag 1 AND tag 2 AND category 3
?tags_and=1,2&categories_and=3,4 tag 1 AND tag 2 AND category 3 AND category 4

You can also use the relation parameter to allow you to make OR queries across multiple taxonomies.

URL Tax Query Matches
?tags=1&categories=3&relation=OR tag 1 OR category 3
?tags=1,2&categories=3&relation=OR tag 1 OR tag 2 OR category 3
?tags=1,2&categories=3,4&relation=OR tag 1 OR tag 2 OR category 3 OR category 4

Or you can combine the two together to make very complex tax queries.

URL Tax Query Matches
?tags_and=1,2&tags=3&relation=OR (tag 1 AND tag 2) OR tag 3
?tags_and=1,2&categories=3&relation=OR (tag 1 AND tag 2) OR category 3
?tags_and=1,2&categories_and=3,4&relation=OR (tag 1 AND tag 2) OR (category 3 AND category 4)
?tags_and=1,2&categories_exclude=3,4&relation=OR (tag 1 AND tag 2) OR ( NOT category 3 OR category4 )

You can even use the relation parameter for queries within a single taxonomy that use multiple types of operators:

URL Tax Query Matches
?tags_and=1,2&tags=3,4&relation=OR (tag 1 AND tag 2) OR tag 3 OR tag 4
?tags_and=1,2&tags_exclude=2,3&relation=OR (tag 1 AND tag 2) OR ( NOT tag 2 OR tag 3 )

I added several additional tests, but with so many new query combination possibilities (it goes from 4 to 18 possible combinations when querying 2 taxonomies), it might help to have a bit more coverage.

@earnjam
6 years ago

#10 @earnjam
6 years ago

I split out the relation parameter code and tests into #44326 because in hindsight it was clear it needed to be it's own ticket.

41287.2.diff just has the code and tests for {$tax}_and parameter support mentioned above.

#11 @Clorith
6 years ago

Just had someone ask about this, which is how I stumbled in here.

To me it makes sense to use 1+2 to get an AND effect, where 1,2 is the OR equivalent. This is how we do it everywhere else, so it makes sense to stick with the established format for, changing it seems counter intuitive to me, unless I'm missing something else here?

#12 @nando99
6 years ago

If I'm reading this right, &categories_and should work... but its not? Will it only work with tags_and?

Last edited 6 years ago by nando99 (previous) (diff)

#13 @earnjam
6 years ago

  • Owner set to earnjam
  • Status changed from new to assigned

@nando99 It should work with any taxonomy in the patch I uploaded, but that was 9 months ago, so it may need a refresh. Core does not currently support AND type queries within any single taxonomy.

@Clorith 1+2 format currently works the same way as 1,2, so changing that would break backwards compatibility.

This ticket was mentioned in Slack in #core-restapi by earnjam. View the logs.


5 years ago

#15 @TimothyBlynJacobs
4 years ago

I think this approach makes sense, though I'm not a huge fan of the name. Would somethings like all_tags make sense? Or am I misunderstanding what this actually does?

#16 @TimothyBlynJacobs
4 years ago

  • Keywords needs-patch needs-unit-tests added; has-patch has-unit-tests removed

5.7 introduces support for an object syntax for term queries to allow for specifying include_children. See #39494. We could extend syntax to have an operator field.

@earnjam would you be interested in working on this for 5.8?

#17 @TimothyBlynJacobs
4 years ago

#52468 was marked as a duplicate.

#18 @stefanolsenn
4 years ago

Any news on this, or workarounds for this? Any help is appreciated!

Version 0, edited 4 years ago by stefanolsenn (next)

#19 @TimothyBlynJacobs
4 years ago

  • Milestone changed from Future Release to 5.8
  • Owner changed from earnjam to dlh

@stefanolsenn we're going to try and take a crack at this for 5.8. Assigning to @dlh.

This ticket was mentioned in Slack in #core by lukecarbis. View the logs.


4 years ago

#21 @helen
4 years ago

@TimothyBlynJacobs @dlh Have you had a chance to take a look at this? With the bugs-only beta period starting May 25 this needs to land very soon if it's going to be for 5.8.

This ticket was mentioned in Slack in #core-restapi by helen. View the logs.


4 years ago

#23 @dlh
4 years ago

I'm aiming to submit a PR for this tomorrow.

#24 @dlh
4 years ago

  • Keywords has-patch has-unit-tests added; needs-patch needs-unit-tests removed

PR submitted: https://github.com/WordPress/wordpress-develop/pull/1249

The patch uses an enum of AND/OR, rather than AND/IN as used by WP_Tax_Query, for general readability and consistency with tax_relation, but that's easy to change if desired.

The patch doesn't include EXISTS in the enum only as it wasn't requested here, but I suppose that it could.

I'm open to feedback about those choices or any others in the PR.

#25 @TimothyBlynJacobs
4 years ago

This looks great to me, thanks @dlh! The only thing I think we should change is to lowercase the AND and OR. That way it matches what we do for things like order which accepts asc instead of ASC.

#26 @desrosj
4 years ago

  • Keywords commit added

Spoke with @TimothyBlynJacobs, and he is going to get this committed in the next few hours for 5.8.

#27 @TimothyBlynJacobs
4 years ago

I missed that we're actually already using the uppercase version for the tax_relation, so I'm just going to ship this as is.

#28 @TimothyBlynJacobs
4 years ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 51026:

REST API: Add support for modifying the term relation when querying posts.

By default, a post most contain any of the requested terms to be included in the response. This commit adds a new operator property that can be set to AND to require a post to contain all of the requested terms.

For example, /wp/v2/posts?tags[terms]=1,2,3&tags[operator]=AND will return posts that have tags with the ids of 1, 2, and 3.

Props dlh, earnjam, Clorith, jnylen0, sebbb.
Fixes #41287.

#29 @milana_cap
4 years ago

  • Keywords needs-dev-note added
Note: See TracTickets for help on using tickets.