Add caching to count_user_posts()

[johnjamesjacoby]

The count_user_posts() function does not cache the results of its query.

On the surface, this seems easy enough, but since it accepts multiple parameters and intersects with page/post editing, it starts to get complex quickly as authors change, posts are transitioned from public to private, and custom post types do more elaborate things.

Since some themes use this directly (Twenty Fourteen, et al) and the REST API exposes this data too, there's at least a small win from not hitting the database for each and every check.

Eventually, count_many_users_posts() may be able to check for cached counts for existing users, and only query for (and prime the caches of) users who do not already have cached counts.

[DJPaul]

We're seeing this uncached function be an issue on a A8C VIP hosted site when querying the REST API users endpoint.

[milindmore22]

Cache for count_user_posts

[milindmore22]

Yes, performances of websites using REST is highly affected.

I have added a patch based on WP VIP function below.

<?php

/**
 * Cached version of count_user_posts, which is uncached but doesn't always need to hit the db
 *
 * count_user_posts is generally fast, but it can be easy to end up with many redundant queries
 * if it's called several times per request. This allows bypassing the db queries in favor of
 * the cache
 */
function wpcom_vip_count_user_posts( $user_id ) {
    if ( ! is_numeric( $user_id ) ) {
        return 0;
    }

    $cache_key = 'vip_' . (int) $user_id;
    $cache_group = 'user_posts_count';

    if ( false === ( $count = wp_cache_get( $cache_key, $cache_group ) ) ) {
        $count = count_user_posts( $user_id );

        wp_cache_set( $cache_key, $count, $cache_group, 5 * MINUTE_IN_SECONDS );
    }

    return $count;
}

Do we need proper cache invalidation or 5 Min expiration timeout will do ?

I think we should handle this in 5.0 thoughts ?

[RogerTheriault]

Just came here to chime in that this (or a similar) improvement is still needed for sites that have many rows in the posts table and use the REST API.

[desrosj]

Changing the version to 3.0 because that's the first version the function existed without caching.

Looking at 39242.diff​, it doesn't take into account the $post_type or $public_only parameters.

Also, looking at other cached values in core, it seems that not specifying an expiration in favor of invalidating the cache when appropriate is preferred. In this case, when a post is updated (published/unpublished, or visibility is changed).

39242.2.diff​ adds the post type to the cache key, and introduces an additional group for when $public_only is true.

I toyed around with where to invalidate caches for a while. Somewhere inside clean_user_cache() feels like the right spot. But there is no way to pass the two additional parameters. Inside wp_insert_post() should be more feasible.

[johnbillion]

This would benefit greatly from tests that cover repeated calls, cache invalidation, etc.

[hellofromTonya]

Hey @whyisjake, With beta 1 coming fast, is this ticket still on your radar for 5.6 milestone? If yes, what's the next step for it?

[hellofromTonya]

In today's core scrub, decision made to punt this ticket to Future Release as 5.6 Beta 1 is 5 days (next Tuesday 20th Oct) and this ticket needs tests.

If any maintainer or committer feels this can be resolved in time, or wishes to assume ownership during a specific cycle, feel free to update the milestone accordingly.

[TimothyBlynJacobs]

#46249 was marked as a duplicate.

[Krstarica]

REST API is still terribly slow, is it possible to fix this issue? Thanks.

[johnregan3]

Not fully functional. Unit tests added, but others are failing.

[johnregan3]

Modified the code so that if an array is passed as the $post_type param, the cache key name will be generated correctly.

I also composed initial Unit Tests.

However, the tests currently come back as such:

There were 3 failures:

1) Tests_Media::test_wp_get_attachment_image_should_use_wp_get_attachment_metadata
Failed asserting that two strings are identical.
--- Expected
+++ Actual
@@ @@
-'<img width="999" height="999" src="http://example.org/wp-content/uploads/2022/07/test-image-testsize-999x999.jpg" class="attachment-testsize size-testsize" alt="" decoding="async" loading="lazy" srcset="http://example.org/wp-content/uploads/2022/07/test-image-testsize-999x999.jpg 999w, http://example.org/wp-content/uploads/2022/07/test-image-large-150x150.jpg 150w" sizes="(max-width: 999px) 100vw, 999px" />'
+'<img width="999" height="999" src="http://example.org/wp-content/uploads/2022/07/test-image-testsize-999x999.jpg" class="attachment-testsize size-testsize" alt="" decoding="async" loading="lazy" srcset="http://example.org/wp-content/uploads/2022/07/test-image-testsize-999x999.jpg 999w, http://example.org/wp-content/uploads/2022/07/test-image-large-1-150x150.jpg 150w" sizes="(max-width: 999px) 100vw, 999px" />'

/var/www/tests/phpunit/tests/media.php:2646

2) WP_Test_REST_Users_Controller::test_get_item_published_author_pages
Failed asserting that 401 is identical to 200.

/var/www/tests/phpunit/tests/rest-api/rest-users-controller.php:3128
/var/www/tests/phpunit/tests/rest-api/rest-users-controller.php:1152

3) Tests_User::test_count_many_users_posts
Failed asserting that '1' matches expected 2.

/var/www/tests/phpunit/tests/user.php:573

I'll continue tracking down these issues, but feel free to jump in and add feedback or your own ideas.

[johnregan3]

The reason unit test # 2 above is failing is because a post is being created and the value returned by count_user_posts() is returning the cached (former) value.

Looks like this will need an action hooked into (probably) 'save_post' that clears this cache. However, as the cache key is dependent upon the post type(s) are requested, it's trickier to guess the precise cache key we need to delete when a post is saved.

I'd love some input on the best way to approach this.

[Krstarica]

We have $cache_key = "count_user_{$post_type_label}_{$userid}"; plus taking into account $public_only param.

On save_post we actually don't know if post_type or user_is has changed, so the safest would be to delete cache for all count_user_% keys, although wp_cache_delete function does not support wildcards.

[wpgurudev]

1. I applied the patch from @johnregan3 , I think the overall approach is fine. The only issue seems that it is not handling certain events which may need to invalidate this cache and recalculate the count.

2. IMO, we can introduce an additional parameter $force = false to count_user_posts function as the last parameter with default value false to be backward compatible.

3. If $force is set to true, function would recalculate the user posts count and set it as a new entry in the cache.

4. This will be useful because we can simply call count_user_posts function with $force set to true on several hooks like save_post, deleted_post etc. to recalculate user posts count. This will further abstract the entire logic of calculating the user posts count without need to know the cache key and group names.

Further Challenges

1. The main issue which I can observe is when we have multiple post types. Consider a situation where we pass

array( 'post', 'page', 'product' )

as $post_type to count_user_posts function.

2. Now, consider if we delete a product then we need to update the cache count_user_post_page_product, but the challenge is how do we identify this combination, because we can easily update count_user_product, but we cannot be sure about post_page_product combination as it can be anything like post_product, page_product etc. as well.

[wpgurudev]

Refactor count_user_posts function

[wpgurudev]

I have refactored the count_user_posts function to two smaller functions

1. count_user_posts_for_single_type: checks count of posts for a single post type. This function leverages the wp cache.

2. count_user_posts_for_multiple_types to check count of posts for multiple post types. This iterates over each post type and calls count_user_posts_for_single_type function in turn.

In addition to this, I have also added clear_count_user_posts_cache function which clears the cache. This function can be called on several conditions like author change of posts, visibility change etc.

I have added one such instance where author change will clear the cache. Some other instances can include

1. Post visibility change from public to private or vice-versa.
2. User deletion.
3. User role change.

and so on..

We can keep adding functions and hook them to appropriate actions to clear the cache using clear_count_user_posts_cache function.

As of now, I'm clearing both the public and non-public posts count cache to keep things simplified.

[wpgurudev]

Refactored function with unit tests

This is just a refresh of the previous patches & tests on the ticket. More work & testing is needed though.

Trac ticket: https://core.trac.wordpress.org/ticket/39242