WordPress.org

Make WordPress Core

Opened 3 years ago

Last modified 4 weeks ago

#21760 reopened enhancement

get_term_by() calls are not cached

Reported by: wonderboymusic Owned by: wonderboymusic
Milestone: Future Release Priority: normal
Severity: normal Version: 2.3
Component: Taxonomy Keywords: needs-patch 2nd-opinion
Focuses: Cc:

Description (last modified by wonderboymusic)

get_term() is the simplest way to retrieve one term, but it requires term_id and taxonomy. Because of this, terms are cached with term_id as key and $taxonomy as bucket. As a result, you can't easily grab a term by slug, unless you use get_term_by( 'slug' ). get_term_by( 'slug' ) and get_term_by( 'name' ) don't even have a query cache, so they go to the database every time. Because you can't get a term by slug without hitting the db, every place you want to get a term by slug: you first have to transform it into a term_id where it will then be cached. This is inefficient because the user may query by slug constantly and never by term_id.

Attachments (20)

get-term-mixed.diff (11.5 KB) - added by wonderboymusic 3 years ago.
tests-posts-tax-cache-group.diff (1.3 KB) - added by wonderboymusic 3 years ago.
term-cache.diff (13.0 KB) - added by wonderboymusic 2 years ago.
term-cache-group.diff (549 bytes) - added by wonderboymusic 2 years ago.
21760.diff (10.8 KB) - added by wonderboymusic 2 years ago.
21760.2.diff (9.8 KB) - added by ryan 2 years ago.
Refresh patch, don't deprecate list_terms_exclusions, back compat cache keys
21760.patch (7.7 KB) - added by johnjamesjacoby 22 months ago.
21760.3.diff (2.6 KB) - added by wonderboymusic 8 months ago.
21760.4.diff (3.0 KB) - added by wonderboymusic 8 months ago.
21760.5.diff (5.2 KB) - added by wonderboymusic 8 months ago.
21760.6.diff (5.0 KB) - added by wonderboymusic 8 months ago.
21760.7.diff (6.8 KB) - added by wonderboymusic 8 months ago.
21760.8.diff (6.9 KB) - added by tollmanz 8 months ago.
s/$bucket/$group and hash name keys
21760.9.diff (7.0 KB) - added by tollmanz 8 months ago.
Add notion of forcing the incrementor update
21760.10.diff (9.2 KB) - added by tollmanz 8 months ago.
21760.2.patch (10.5 KB) - added by boonebgorges 5 months ago.
21760.11.diff (2.2 KB) - added by nacin 5 months ago.
I don't see how this is any less DRY. It is certainly more obvious as to what is happening.
21760.12.diff (5.1 KB) - added by tollmanz 4 months ago.
21760.13.diff (4.1 KB) - added by tollmanz 4 months ago.
21760.14.diff (10.0 KB) - added by boonebgorges 3 months ago.

Download all attachments as: .zip

Change History (105)

comment:1 @nacin3 years ago

Cache keys are cast to strings. That means there would be a collision of the term_id of 1, the slug of '1', and the name of '1'. Even without the casting woes, slug and name would conflict.

For users, we handle this with separate cache buckets: users (by ID), userlogins, useremail, and userslugs.

This can just as easily be handled in the same bucket with prefixes - IDs are stored as is, login-$login, email-$email, slug-$slug. Or for terms, you'd have IDs, slug-$slug, name-$name.

update_term_cache() should set slug and name cache keys, in addition to ID, and that should then be leveraged by get_term_by() and friends.

comment:2 @nacin3 years ago

See also #11531.

We should probably rename these buckets. Rather than $taxonomy and {$taxonomy}_relationships, we should use something that isn't valid in taxonomy/CPT name. taxonomy:$taxonomy and relationships:$taxonomy seem sufficient. These are the only buckets I know of that have a dynamic component to them.

comment:3 @wonderboymusic3 years ago

Per chat with Nacin - gonna work on separate cache buckets for each type and will resurrect get_term_by() which can leverage them. Also, get_term() would have a hard time distinguishing between id, name, and slug for the category named, slugged, term_id'd "1" etc

comment:4 @scribu3 years ago

  • Keywords needs-patch added; has-patch removed

comment:5 @scribu3 years ago

  • Summary changed from Allow get_term() to accept string as first argument to get_term_by() calls are not cached

comment:6 @dd323 years ago

FWIW, by overloading get_term( $id ... to accept int|string|object, it brings up conditions where

  • Slug is numeric
  • a term_id is passed in, but return term_ids as strings in some locations, so you can't rely upon type being an int for term_ids

comment:7 @wonderboymusic3 years ago

Attached a new patch:

1) get_term_by() calls are cached
2) term_cache_group() abstracts cache bucket for terms
3) update_term_cache() adds to cache of all 3 buckets
4) delete_term_cache() deletes cache of all 3 buckets
5) get_terms() calls update_term_cache(), deprecates the buggy list_terms_excusions in favor of get_terms filter
6) Lots of cleanup and streamlining of clean_term_cache() calls

comment:8 @scribu3 years ago

  • Keywords has-patch needs-unit-tests added; needs-patch removed

This looks like an awesome patch, but it's tiring to read, due to all the whitespace fixes. I'll try again in the morning.

In related news, the test suite doesn't seem to have a single call to get_term_by( 'slug', ... );

comment:9 @scribu3 years ago

Also, the patch seems to fail an existing test:

There was 1 failure:

1) Tests_Term::test_set_object_terms_by_name
Failed asserting that '1' matches expected 5.

/home/scribu/svn/wp-tests/tests/term.php:135

Finally, we get some use out of these darn things. :)

comment:10 @wonderboymusic3 years ago

Updated the patch - the problem was a $wpdb->prepare thing in the IN query in clean_term_cache(). All tests pass now.

WITH ONE CAVEAT:
tests-posts-tax-cache-group.diff needs to be applied to Unit Tests so that the Post tests read from the proper term cache group

comment:11 @wonderboymusic3 years ago

  • Description modified (diff)

comment:12 @scribu3 years ago

Right, except term_cache_group() is introduced in your main patch. :)

In this case, just add additional test cases to tests-posts-tax-cache-group.diff

comment:13 @scribu3 years ago

Also, I'm not sure term_cache_group() should be a public function; maybe rename it to _term_cache_group() and mark it as @private.

comment:14 @ryan3 years ago

Instead of caching multiple copies of an object, we usually cache IDs that can be used to look up a single cached copy. slug -> ID, ID -> object.

I'm not really digging term_cache_group(). I consider cache keys and groups public API that shouldn't change.

comment:15 @nacin3 years ago

Some thoughts:

  • The code in delete_term_cache() seems like it belongs in clean_term_cache(). (This code is similarly in clean_user_cache().)
  • Not *entirely* sure that term_cache_group() is necessary. It only gets called in get_term(), get_term_by(), get_terms(), and a few relationship functions. Doing "taxonomy:$taxonomy:slugs" etc. seems pretty safe, and not unreasonably more complicated (or longer) than _term_cache_group(). Though... we might need to special-case the core taxonomies to avoid renaming those buckets, as ryan pointed out.
  • With user login, slug, and email buckets, we don't cache the user object, but rather, just the ID. That way, we can then consult the ID-based bucket for the object. We should do the same here. See WP_User::get_data_by() for how that works.
  • Removing list_terms_exclusions should probably come in the separate ticket, #21267.
  • In get_cat_ID(), you changed if ( $cat ) to an is_object() check. Is that necessary? Will get_term_by() ever turn a value that evaluates to true that is *not* an object?
  • I had to do an svn diff -x --ignore-all-space to be able to review the patch effectively. Whitespace changes, especially in functions not quite touched by the patch, make things a bit more difficult to review.

comment:16 @wonderboymusic3 years ago

Updated patch to only cache IDs for slug and name - sorry about the whitespace, some of the term functions are esoteric and hard to read without some daylight. I can revert those changes in a subsequent .diff

My reasoning for term_cache_group was 1) standardizing group generation 2) flexibility in making changes in the future, only has to be done in one place. If this is deemed unnecessary, is it to retain the old groups?

comment:17 @wonderboymusic3 years ago

Another piece of this - get_terms calls update_term_cache if you got all the fields, regardless of checking the hide_empty arg.

$terms = $wpdb->get_results( $query );
if ( 'all' == $fields ) {
	update_term_cache( $terms );
}

I did a performance test where I created 20,000 tags and called get_terms( 'post_tag', array( 'hide_empty' => false ). This will create 60,000 cache entries which exhausted memory until I bumped up to 256M. I tested posts with 1000 tags and they seemed fine, but this one may need to exclude terms with count of 0 before calling update_term_cache with every returned term

comment:18 @wonderboymusic2 years ago

updated the patch:

1) removed all of my whitespace fixes (that was a gift from heaven, let's just say I learned my lesson)
2) restored all return-by-reference things
3) moved delete_term_cache code to clean_term_cache

The patch is WAY more light now - next up is removing term_cache_group as per Boren's feedback

comment:19 @nacin2 years ago

Why were the return-by-references added back? I wonder if we still need them anymore.

We might need to keep term_cache_group() as we'll need to keep the standard category, post_tag, and link_category buckets, but we should be able to get away with prefixing the buckets of any custom taxonomy.

@wonderboymusic2 years ago

comment:20 @wonderboymusic2 years ago

return-by-references need to die everywhere, they are PHP4 fluff. I just added them back because I was trying to tighten to scope of my changes. The only scenario where they make sense to use is $a =& $b, not $a =& my_function();

Patch has been updated with ret-by-refs removed.

"we'll need to keep the standard category, post_tag, and link_category buckets" - I took a stab at this, does it accomplish what you want?

comment:21 @bpetty2 years ago

  • Keywords needs-refresh added

I don't feel very comfortable with refreshing this patch myself with the latest changes, but I'll be happy to jump back in and do some testing if someone else can jump on this quickly for 3.5.

comment:22 @scribu2 years ago

  • Milestone changed from Awaiting Review to Future Release

Considering it's an enhancement, the patch isn't trivial, it doesn't have unit tests yet and that we're 5 weeks away from release, it's not for 3.5.

comment:23 @aaroncampbell2 years ago

  • Cc aaroncampbell added

@wonderboymusic2 years ago

comment:24 @wonderboymusic2 years ago

  • Keywords needs-refresh removed
  • Milestone changed from Future Release to 3.6

21760.diff Patch refreshed for trunk, passes all Unit Tests.

term-cache-group.diff Patch for Unit Tests so term_cache_group() works. i recall Ryan hating term_cache_group() - I am very open to suggestions on it, but I like it.

Last edited 2 years ago by wonderboymusic (previous) (diff)

comment:25 @husobj2 years ago

  • Cc ben@… added

comment:26 follow-up: @ryan2 years ago

I don't mind the function. I mind changing cache locations. There is too much plugin code that directly accesses the current cache locations for both builtin and custom taxonomies.

list_terms_exclusions shouldn't be deprecated. I see too many plugins using it.

@ryan2 years ago

Refresh patch, don't deprecate list_terms_exclusions, back compat cache keys

comment:27 in reply to: ↑ 26 @nacin2 years ago

Replying to ryan:

I don't mind the function. I mind changing cache locations. There is too much plugin code that directly accesses the current cache locations for both builtin and custom taxonomies.

At that point, the function isn't needed. But, would be nice to namespace these as much as possible (and thus keep the function)? The only existing buckets are $taxonomy and {$taxonomy}_relationships. Maybe new buckets can all be taxonomy:$taxonomy:$group. I'd personally love to escape the generic $taxonomy bucket, at least for custom taxonomies, as the problem there is gonna continue to exist — #11531 — but I'll take what I can get.

comment:28 @nacin23 months ago

  • Milestone changed from 3.6 to Future Release

This is cool but let's review it early 3.7.

comment:29 @johnjamesjacoby22 months ago

Late to this party, but already wrote up a patch based on a function we're using for WordPress.com VIP's.

Patch also cleans up get_term()

@johnjamesjacoby22 months ago

comment:30 @ethitter22 months ago

  • Cc erick@… added

comment:31 @wonderboymusic19 months ago

  • Owner set to johnjamesjacoby
  • Status changed from new to assigned

Assigning to JJJ to investigate his patch

comment:32 @Chouby18 months ago

  • Cc frederic.demarle@… added

comment:33 @nacin17 months ago

Can someone step up to opine on this? How are we doing for unit tests in this area? Would be good to make sure there are some tests to assert there is no cache pollution.

comment:34 @nacin17 months ago

#11531 was marked as a duplicate.

comment:35 @wonderboymusic11 months ago

  • Keywords needs-refresh added
  • Milestone changed from Future Release to 4.0
  • Owner changed from johnjamesjacoby to wonderboymusic

this should definitely be tackled as part of our Taxonomy work in 4.0 - I will take another stab at this

comment:36 @helen8 months ago

wonderboymusic - speak/unit test now or punt from 4.0, leaving it to you.

comment:37 @wonderboymusic8 months ago

I will either look at this with gusto in the next few days or punt with gusto

@wonderboymusic8 months ago

comment:38 @wonderboymusic8 months ago

  • Keywords needs-refresh removed

I'm gonna do some profiling/testing, but 21760.3.diff should do it for get_term_by( 'slug', ... ).

@wonderboymusic8 months ago

comment:39 @wonderboymusic8 months ago

21760.4.diff is better.

Unit test query counts

Install (before patch): 403
Install (after patch): 384

Test suite (before patch): 208474
Test suite (after patch): 208257

@wonderboymusic8 months ago

comment:40 @wonderboymusic8 months ago

  • Keywords needs-unit-tests removed

21760.5.diff adds unit tests

@wonderboymusic8 months ago

comment:41 @wonderboymusic8 months ago

21760.6.diff is a simpler approach and passively invalidates slugs via an increment on the bucket. In my previous attempts, clean_term_cache() ended up producing 1000-2000 more database queries when running the entire test suite. No bueno.

@wonderboymusic8 months ago

comment:42 @wonderboymusic8 months ago

After adding entries for slug AND name in 21760.7.diff:

Test suite (after patch): 208,212 (-262 database calls)

The homepage of my test install makes 6 less queries.

Added unit tests for get_term_by( 'name', .... )

comment:43 @wonderboymusic8 months ago

  • Keywords commit added

I like this, but I will throw some cold water on my face and wait

comment:44 @tollmanz8 months ago

Nice work! I love the idea of wp_get_last_changed(). An incrementor API like this would be most excellent in core.

A couple thoughts on the patch:

  • Line 980 of src/wp-includes/taxonomy.php should use $term->term_id as the key, not the $term object.
  • $bucket is not a term used in core (other than fleetingly in a few comments). I think for clarity and consistency, we should stick with the $group nomenclature.
  • The keys for the term names will prove problematic. Whitespace is not allowed in Memcached keys. Additionally, these keys can only be a grand total of 250 characters in Memcached (and I'm assuming there are other limits in other object caches). These should be hashed or sanitized in someway before being used as keys.
  • Playing devil's advocate, if a term is not found in get_term_by(), should that result still be cached to avoid the redundant look up in the future? I think that would be nice, but complicates checking for the "non-existent" values, as well as the invalidation of that cache.
  • I don't think update_term_cache() will work. 1) Using wp_cache_add() will fail to save new data as it is already set. A quick and dirty fix for this is to use wp_cache_set(). It'll guarantee new data; however, there is a slight performance penalty. 2) Since we are using incrementors, we should...well...use them. We should generate a new incrementor and save the new values with the new incrementor. This causes a cache miss on the old incrementor and saves the new one. This could be accomplished by adding a $force parameter to wp_get_last_changed(). If set to true, it would not pull the incrementor from cache and instead regenerate it and save over the old one. Alternatively, it could just erase the old one.

Finally, as excited as I am about the incrementor idea, I do have some concerns. Incrementors work great with caches that use an LRU eviction policy (e.g., Memcached). You can fairly carelessly add incrementors to cause cache misses/invalidations. This works well because the old incrementors are no longer accessed and eventually fall off the slab. With some caching systems, the eviction policy is a bit problematic for an incrementor system (e.g., APC fills the cache until full, them dumps everything). We need to be really careful about how we do this to make sure it will consistently work across object caches. Ideally, when we change the incrementor, we delete the old one or overwrite it.

I think we need to be very careful as we implement this. An incrementor system can be extremely powerful, yet also really dangerous. Perhaps we should start a separate ticket to explore this? If we were to implement this in core, it would be great to have a more robust system for handling this with methods for setting, getting, and invalidating incrementors.

Last edited 8 months ago by tollmanz (previous) (diff)

comment:45 @tollmanz8 months ago

Scratch my comment about using $term->term_id instead of $term. I see that $term is changed from the object into the term ID earlier in the code.

@tollmanz8 months ago

s/$bucket/$group and hash name keys

@tollmanz8 months ago

Add notion of forcing the incrementor update

comment:46 @tollmanz8 months ago

21760.8.diff and 21760.9.diff change the $bucket nomenclature to $group, hash the term name key, and add a way to invalidate the incrementor. The first two changes are in .8 and all three are in .9.

comment:47 @rmccue8 months ago

Psst, while we're in the term caching arena, #28743 (especially if we want to hit clean_term_cache anyway).

@tollmanz8 months ago

comment:48 @tollmanz8 months ago

21760.10.diff iterates on this patch by adding /updating the following:

  • I found a slight bug in the get_term_by() adds the $term object to the cache. The values were cached before the $term object was run through the sanitization function. The sanitization function converts ints to strings. Therefore, a term grabbed from the cache would be different than a term grabbed directly from MySQL since the term grabbed from cache would never be sanitized. I updated two of the unit tests to specifically test for this.
  • I added suspended cached invalidation awareness to address #28743.
  • A unit test was added to test #28743.

Thoughts/questions:

  • I'm starting to think that we shouldn't add the if ( $cache ) conditional. If the value is cached, the filters and sanitization functions are completely missed. I can see an argument that the terms will have already been run through the filters and sanitization routines; however, this would cause problems if a user, using a persistent cache, installs a plugin that should further manipulate these terms. The user would need to wait until the next cache invalidation for it to take effect. These things cause devs to go batty.
  • We should probably add suspended cached invalidation awareness to update_term_cache(), right? I looked at how this is handled in update_post_cache() and it is not aware of suspended cached invalidation; however, that is because the function is not an invalidation function. It will add items to cache, but does not overwrite it or delete cached items. It will never do this. update_term_cache() operates differently. It nukes the cached items and forces an update. As such, it should respect suspended cached invalidation, right?

comment:49 @wonderboymusic8 months ago

tollmanz: good stuff, waiting for feedback from @nacin

comment:50 @ircbot7 months ago

This ticket was mentioned in IRC in #wordpress-dev by DrewAPicture. View the logs.

comment:51 @ocean907 months ago

  • Keywords 4.1-early added
  • Milestone changed from 4.0 to Future Release

@boonebgorges5 months ago

comment:52 @boonebgorges5 months ago

  • Keywords commit 4.1-early removed
  • Milestone changed from Future Release to 4.1

21760.2.patch does the following:

  • refresh for current trunk (whitespace was preventing from applying)
  • I like tollmanz's suggestion https://core.trac.wordpress.org/ticket/21760#comment:48 that cached values get returned through the filters, so it's implemented here
  • I also think he's correct that cache invalidation preferences should be respected in update_term_cache(), so it's in there too
  • The change in update_term_cache() that forces 'last_changed' to be incremented caused a failing test related to get_terms(). It turns out that get_terms() is calling update_term_cache() whenever you call it with 'fields=all'. This looks like a bug to me, since the terms are properly cached after the queries are run later in the function (and that bit of code goes back to Ye Olden Days of WP). I've removed it.

With some caching systems, the eviction policy is a bit problematic for an incrementor system (e.g., APC fills the cache until full, them dumps everything). We need to be really careful about how we do this to make sure it will consistently work across object caches. Ideally, when we change the incrementor, we delete the old one or overwrite it.

FWIW, I think this is only the default behavior. I'm pretty sure that if you set a non-zero ttl, only expired items are dumped when the cache fills. More generally, while I totally agree that we should constantly be mindful of the realities of various caching systems, it's also reasonable to expect that the people implementing those systems will be cognizant of the requirements of the particular backend they've chosen, and will configure it accordingly.

comment:53 @wonderboymusic5 months ago

  • Resolution set to fixed
  • Status changed from assigned to closed

In 29915:

Cache get_term_by() calls:

  • Add a helper function, wp_get_last_changed(), to retrieve a last-modified timestamp by cache group
  • When caching a term, also make cache entries for slug and name via slug:{$term_id} and name:{$term_id} keys in the $taxonomy:$last_changed bucket that reference the term_id
  • In clean_term_cache() and update_term_cache(), respect $_wp_suspend_cache_invalidation
  • Original term cache entries maintain BC

Adds unit tests.

Props wonderboymusic, tollmanz, boonebgorges.
Fixes #21760.

comment:54 @wonderboymusic5 months ago

My description of those cache keys was wrong....

slug:{$term->slug} and name:md5( $term->name )

Last edited 5 months ago by wonderboymusic (previous) (diff)

comment:55 @nacin5 months ago

This looks great, but I'm reopening for some potential tweaks:

  • wp_get_last_changed() is really weird. I am not against helpers but in this case it seems to do too many things and is generally confusing to see. It gets the value but sometimes sets the value? The name is wrong. As it stands I'd eliminate it.
  • This suddenly adds new and arbitrary groups. That's not really good. We should have individual buckets for slugs and names (as done in users), potentially with generic names (like 'termslugs' and 'termnames' ).
  • Why the md5? Key length? Whitespace?

comment:56 @nacin5 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:57 follow-up: @wonderboymusic5 months ago

Why the md5? Key length? Whitespace?

Whitespace is not allowed in Memcached keys

wp_get_last_changed() is mostly for DRY - open to suggestions on that one, I couldn't think of a better name

comment:58 in reply to: ↑ 57 @nacin5 months ago

Replying to wonderboymusic:

Why the md5? Key length? Whitespace?

Whitespace is not allowed in Memcached keys

This isn't our problem, per [27300]. Plenty of other areas of core could be affected by this. For example, who is to say a slug can't have a space?

@nacin5 months ago

I don't see how this is any less DRY. It is certainly more obvious as to what is happening.

comment:59 @boonebgorges4 months ago

#28743 was marked as a duplicate.

@tollmanz4 months ago

comment:60 @tollmanz4 months ago

21760.12.diff combines the changes @nacin made with his other suggestions, which include:

  • Removing md5 hashes (seems to be consistent with how similar strings are handled in WP)
  • Attempts to use a more sane grouping convention. It borrows from the user caching and adds the incrementor to the key
  • I also noticed that the incrementor was only used with the name and slug caches. I added the incrementor to the term object cache as well, but perhaps I am missing some reason why this isn't a good idea.


comment:61 @wonderboymusic4 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 30073:

Adjust caching for get_term_by() calls:

  • Remove md5 hashes for term name cache keys
  • Remove the namespace for the keys for names and slugs and add them to the group names
  • Remove wp_get_last_changed(), which @nacin hated


Props tollmanz.
Fixes #21760.

comment:62 @boonebgorges4 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

This has broken many things. $ phpunit --group taxonomy

comment:63 @boonebgorges4 months ago

In 30080:

Remove call to wp_get_last_changed() from unit tests.

This function was removed in [30073].

See #21760.

comment:64 @boonebgorges4 months ago

[30080] remove the fatal error so that at least the tests can continue to run. A number are still failing.

comment:65 @tollmanz4 months ago

I ran the whole unit test suite before adding these changes because I was worried about such breakage. I saw no failures, but I think this was because I was running them wrong. When running phpunit --group taxonomy, does it run different tests than what is run with phpunit? I know it will run a smaller subset, but wondering if it also includes ones that wouldn't be run with phpunit alone.

comment:66 @boonebgorges4 months ago

Ah, it probably skipped these tests because the ticket was open. $ phpunit --group 21760 should do it.

@tollmanz4 months ago

comment:67 @tollmanz4 months ago

21760.13.diff adds the cache group change to one more place that was missed in the previous patch. Additionally, it updates the unit tests to use the cache group change that is used in the update. The update allows the tests pass for this ticket and for the tests in general.

Time to riot!

comment:68 @boonebgorges4 months ago

Let the rioting commence.

comment:69 @boonebgorges4 months ago

  • Resolution set to fixed
  • Status changed from reopened to closed

In 30108:

Clean up get_term_by() caching.

  • Fix cache key/group modification that was missed in [30073].
  • Update unit tests to reflect new key/group format.

Props tollmanz.
Fixes #21760.

comment:70 @boonebgorges4 months ago

In 30112:

Clean up cache invalidation suspension global in unit tests.

This fixes a test that was introduced in [30073] which was polluting later
tests.

See #21760.

comment:71 @slackbot4 months ago

This ticket was mentioned in Slack in #core by jeremyfelt. View the logs.

comment:72 follow-up: @nacin3 months ago

  • Resolution fixed deleted
  • Status changed from closed to reopened

This most likely broke something on .com (whether they know about it yet or not) and in some plugins that mess directly with term caches. But it's not just that.

First, the pedantic stuff I've previously covered: We cannot change the group names like this. If we could, we could have abandoned 'post_tag', 'category', and $taxonomy a long time ago. We can likely change $taxonomy to "taxonomy-$taxonomy" or just 'taxonomy' or 'term' (proposed either here or elsewhere).

(I wrote two years ago on this ticket: "we'll need to keep the standard category, post_tag, and link_category buckets, but we should be able to get away with prefixing the buckets of any custom taxonomy.")

Here's how I would expect last_changed to be used:

  • The term 1, name Apple, slug apple, is saved.
  • A term object is cached with key = 1, value = object, group = 'term' (or similar)
  • A term slug is cached with key = apple, value = 1 (ID), group = 'termslugs' (or similar)
  • A term name is cached with key = 'Apple', value = 1 (ID), group = 'termnames' (or similar)
  • The key last_changed for the group 'term' is set to "0.51245500 1418627718" (microtime())
  • get_terms() caches its queries. When doing so, it includes the value of last_changed in its cache key

Later, this term is updated:

  • The caches for 1 (group term), apple (group termslugs), Apple (group termnames) are deleted
  • The new object, slug, and name are set into cache
  • last_changed is bumped, and get_terms()'s cache is invalidated

As it stands now, whenever any term is edited, every term's cache gets completely wiped out. That seems like it's worse than it was previously. What am I missing?

Additionally, what's in trunk now actually makes a lot of code redundant or useless. What's the point of wp_cache_delete($term->term_id, $term->taxonomy); when that's no longer the name of the group, and any change invalidates everything everywhere? Does clean_term_cache() even work anymore? It doesn't look like it does. That's a problem and it's going to break things.

At this point, I think we need to revert this from the branch and try again in 4.2.

comment:73 @slackbot3 months ago

This ticket was mentioned in Slack in #core by nacin. View the logs.

comment:74 in reply to: ↑ 72 @ericmann3 months ago

Replying to nacin:

This most likely broke something on .com (whether they know about it yet or not) and in some plugins that mess directly with term caches. But it's not just that.

Breaking anything at this stage feels like a sub-optimal situation. I agree with the overall sentiment that we should roll this back, take a step away, and revisit early in 4.2.

Here's how I would expect last_changed to be used: ...

The notes in this workflow actually pan out to some great tests to make sure the code is fairly robust, doesn't break anything, and does exactly what we want it to do. Additionally, we can make sure that the changes to cache don't introduce any breaking changes that affect large installations like .com.

At this point, I think we need to revert this from the branch and try again in 4.2.

Agree. Really wanted to see this ship in 4.1, but it doesn't feel ready.

comment:75 @boonebgorges3 months ago

nacin and ericmann - This sounds sensible to me. The workflow nacin describes sounds like a narrower use of the last_changed iterator, which IMO is a good thing - we should only be invalidating with a sledge hammer when we can't do it with a laser beam.

@boonebgorges3 months ago

comment:76 @slackbot3 months ago

This ticket was mentioned in Slack in #core by boone. View the logs.

comment:77 @boonebgorges3 months ago

In 30900:

Remove caching for get_term_by() calls.

The new cache group scheme causes term invalidation to be overly broad, so
that busting the cache for one term will bust the cache for all terms in the
taxonomy. We'll have another go at more focused use of the 'last_changed'
incrementor in a future release.

Reverts [29915], [30073], [30080], [30108], [30112].
See #21760.

comment:78 @tollmanz3 months ago

Bummer that we didn't catch this earlier :( Thanks for removing this boonebgorges. Hopefully we can figure out a more reasonable patch for 4.2.

comment:79 @johnbillion3 months ago

  • Keywords commit fixed-major added

comment:80 @johnbillion3 months ago

In 30914:

Remove caching for get_term_by() calls.

The new cache group scheme causes term invalidation to be overly broad, so
that busting the cache for one term will bust the cache for all terms in the
taxonomy. We'll have another go at more focused use of the 'last_changed'
incrementor in a future release.

Reverts [29915], [30073], [30080], [30108], [30112].

Merges [30900] to the 4.1 branch.

See #21760.

comment:81 @johnbillion3 months ago

  • Keywords needs-patch 2nd-opinion added; has-patch commit fixed-major removed
  • Milestone changed from 4.1 to Future Release

comment:82 @boonebgorges3 months ago

In 30954:

Update individual term caches in get_terms().

This was removed in [29915] as part of the attempt to add cache support to
get_term_by(). When that support was removed in [30900], it was not properly
restored.

This changeset includes a unit test to verify that the cache is properly primed
for terms found in get_terms(), as well as tests to verify the other cache
setting that was touched by [30900].

Fixes #30749. See #21760.

comment:83 @nacin3 months ago

In 30959:

Update individual term caches in get_terms().

Merges [30954] to the 4.1 branch.

props boonebgorges.
fixes #30749. see #21760.

comment:84 @slackbot2 months ago

This ticket was mentioned in Slack in #core by ericmann. View the logs.

comment:85 @dd324 weeks ago

#27572 was marked as a duplicate.

Note: See TracTickets for help on using tickets.