Sitemap pages 404 with more than one page

[loranrendel]

When there are more than 1 page in sitemap there may be an error: the sitemap will be provided with right content but 404 code.

For example I will decrease max url count from 1000 to 2 in new test WordPress installation with 9 posts:

<?php
add_filter('wp_sitemaps_max_urls', function () {
    return 2;
});

​https://testwp.xpor.org/wp-sitemap-posts-post-1.xml 200
​https://testwp.xpor.org/wp-sitemap-posts-post-2.xml 404

WP_Query dump:

WP_Query Object
(
    [query] => Array
        (
            [paged] => 2
            [sitemap] => posts
            [sitemap-subtype] => post
        )

    [query_vars] => Array
        (
            [paged] => 2
            [sitemap] => posts
            [sitemap-subtype] => post
            [error] => 
            [m] => 
            [p] => 0
            [post_parent] => 
            [subpost] => 
            [subpost_id] => 
            [attachment] => 
            [attachment_id] => 0
            [name] => 
            [pagename] => 
            [page_id] => 0
            [second] => 
            [minute] => 
            [hour] => 
            [day] => 0
            [monthnum] => 0
            [year] => 0
            [w] => 0
            [category_name] => 
            [tag] => 
            [cat] => 
            [tag_id] => 
            [author] => 
            [author_name] => 
            [feed] => 
            [tb] => 
            [meta_key] => 
            [meta_value] => 
            [preview] => 
            [s] => 
            [sentence] => 
            [title] => 
            [fields] => 
            [menu_order] => 
            [embed] => 
            [category__in] => Array
                (
                )

            [category__not_in] => Array
                (
                )

            [category__and] => Array
                (
                )

            [post__in] => Array
                (
                )

            [post__not_in] => Array
                (
                )

            [post_name__in] => Array
                (
                )

            [tag__in] => Array
                (
                )

            [tag__not_in] => Array
                (
                )

            [tag__and] => Array
                (
                )

            [tag_slug__in] => Array
                (
                )

            [tag_slug__and] => Array
                (
                )

            [post_parent__in] => Array
                (
                )

            [post_parent__not_in] => Array
                (
                )

            [author__in] => Array
                (
                )

            [author__not_in] => Array
                (
                )

            [ignore_sticky_posts] => 
            [suppress_filters] => 
            [cache_results] => 1
            [update_post_term_cache] => 1
            [lazy_load_term_meta] => 1
            [update_post_meta_cache] => 1
            [post_type] => 
            [posts_per_page] => 10
            [nopaging] => 
            [comments_per_page] => 50
            [no_found_rows] => 
            [order] => DESC
        )

    [tax_query] => WP_Tax_Query Object
        (
            [queries] => Array
                (
                )

            [relation] => AND
            [table_aliases:protected] => Array
                (
                )

            [queried_terms] => Array
                (
                )

            [primary_table] => wp_posts
            [primary_id_column] => ID
        )

    [meta_query] => WP_Meta_Query Object
        (
            [queries] => Array
                (
                )

            [relation] => 
            [meta_table] => 
            [meta_id_column] => 
            [primary_table] => 
            [primary_id_column] => 
            [table_aliases:protected] => Array
                (
                )

            [clauses:protected] => Array
                (
                )

            [has_or_relation:protected] => 
        )

    [date_query] => 
    [request] => SELECT SQL_CALC_FOUND_ROWS  wp_posts.ID FROM wp_posts  WHERE 1=1  AND wp_posts.post_type = 'post' AND (wp_posts.post_status = 'publish' OR wp_posts.post_status = 'private')  ORDER BY wp_posts.post_date DESC LIMIT 10, 10
    [posts] => Array
        (
        )

    [post_count] => 0
    [current_post] => -1
    [in_the_loop] => 
    [comment_count] => 0
    [current_comment] => -1
    [found_posts] => 0
    [max_num_pages] => 0
    [max_num_comment_pages] => 0
    [is_single] => 
    [is_preview] => 
    [is_page] => 
    [is_archive] => 
    [is_date] => 
    [is_year] => 
    [is_month] => 
    [is_day] => 
    [is_time] => 
    [is_author] => 
    [is_category] => 
    [is_tag] => 
    [is_tax] => 
    [is_search] => 
    [is_feed] => 
    [is_comment_feed] => 
    [is_trackback] => 
    [is_home] => 
    [is_privacy_policy] => 
    [is_404] => 1
    [is_embed] => 
    [is_paged] => 
    [is_admin] => 
    [is_attachment] => 
    [is_singular] => 
    [is_robots] => 
    [is_favicon] => 
    [is_posts_page] => 
    [is_post_type_archive] => 
    [query_vars_hash:WP_Query:private] => bcf5fd65d0a7962d637cd5cb9d865508
    [query_vars_changed:WP_Query:private] => 
    [thumbnails_cached] => 
    [stopwords:WP_Query:private] => 
    [compat_fields:WP_Query:private] => Array
        (
            [0] => query_vars_hash
            [1] => query_vars_changed
        )

    [compat_methods:WP_Query:private] => Array
        (
            [0] => init_query_flags
            [1] => parse_tax_query
        )

)

[peterwilsoncc]

Thank you for your report.

I am able to reproduce this from version 5.5, I'll bring this to the attention of the sitemap maintainers for their review.

[peterwilsoncc]

I think the root cause of this is #51117.

The report in #53095 suggests this is a more serious problem for sites with a custom post type that contains more post objects than there are native WordPress post post objects.

As the sitemaps execute the main (is_home()) query, that query is used to determine the the pages status. On sites with multiple pages in the site map, then the page parameter is passed to the is_home query causing a file not found error if the frontend would not have the same number of pages.

Consider the following site:

• 9 post objects
• 3500 custom post type objects

Page two of the CPT sitemap will 404 as there is only a single page of posts according the the posts per page setting in the dashboard.

The same will apply for user and taxonomy site maps if there are more authors or terms than posts.

[peterwilsoncc]

#53095 was marked as a duplicate.

[tigerfinch]

(Apologies if I'm tagging this wrong... I've tagged has-patch as I've got a potential resolution)

The solution is – when a sitemap is being generated – to force the main query to use the post_type of the sitemap.

As an interim solution for users, I found this worked in theme/plugin code:

  add_filter('pre_get_posts', function($query) {
    global $wp_query;
    if ($wp_query->query['sitemap'] === 'posts')
      $query->set('post_type', $wp_query->query['sitemap-subtype']);
    return $query;
  });

As a core fix for this, we could alter the function function register_rewrites() in class-wp-sitemaps.php to pass the requested custom post type to the main query:

// Register routes for providers.
add_rewrite_rule(
  '^wp-sitemap-([a-z]+?)-([a-z\d_-]+?)-(\d+?)\.xml$',
  'index.php?sitemap=$matches[1]&sitemap-subtype=$matches[2]&paged=$matches[3]',
  'top'
);

to

// Register routes for providers.
add_rewrite_rule(
  '^wp-sitemap-([a-z]+?)-([a-z\d_-]+?)-(\d+?)\.xml$',
  'index.php?sitemap=$matches[1]&sitemap-subtype=$matches[2]&post_type=$matches[2]&paged=$matches[3]',
  'top'
);

[tigerfinch]

Whoops, didn't mean to close the issue, apologies

[Tkama]

I described the problem and temporary solution here ​https://wp-kama.com/handbook/sitemap/bag-404-pagination

The core solution is to move Sitemap init from template_redirect hook to parse_request hook, as it's done for REST API.

[RavanH]

This plugin should resolve the issue ​https://wordpress.org/plugins/xml-sitemaps-manager/ which applies the solution proposed by @Tkama Please let me know if there are still problems after applying the fix.

[mystery8]

any update about this? why this bug gets low priority? It will cause big sites (>2000 pages) lose many pages in serp.

@tigerfinch you should add:

<?php
isset( $wp_query->query['sitemap'] )

or you gonna get: Undefined index: sitemap

[RavanH]

Replying to mystery8:

any update about this? why this bug gets low priority? It will cause big sites (>2000 pages) lose many pages in serp.

Try ​https://wordpress.org/plugins/xml-sitemaps-manager/ it will not change your sitemap, only fix the bug plus give you some options (which you can ignore). You can simply remove the plugin after the bug is fixed in core.

[peterwilsoncc]

#57961 was marked as a duplicate.

[Cybr]

A user is experiencing this error: ​https://wordpress.org/support/topic/sitemap-404-error-15/#post-18049558.

[phanduynam]

currently this damn bug still exists on wordpress 6.8.2 and it seriously affects seo. i don’t understand why this serious bug still exists after so many years.

[dd32]

#64000 was marked as a duplicate.

[phanduynam]

@dd32 the important thing is that no one has addressed the bug and it seems to have been forgotten for years. how could such a serious bug be allowed to exist for so long?

[phanduynam]

Has anyone solved this problem?

[RavanH]

Replying to phanduynam:

Has anyone solved this problem?

Hi @phanduynam, both @tigerfinch and @Tkama provided solutions above. A variant of a solution from @Tkama is implemented in ​https://wordpress.org/plugins/xml-sitemaps-manager/

If you prefer a raw code snippet, try:

<?php
add_action( 'parse_request', 'track51912_sitemaps_loaded' );

function track51912_sitemaps_loaded( $wp ) {
        global $wp_query;

        if ( empty( $wp->query_vars['sitemap'] ) && empty( $wp->query_vars['sitemap-stylesheet'] ) ) {
                return;
        }

        // Prepare query variables.
        $query_vars           = $wp_query->query_vars;
        $wp_query->query_vars = $wp->query_vars;

        // Render the sitemap.
        wp_sitemaps_get_server()->render_sitemaps();

        // Still here? Then it was an invalid sitemap request after all. Undo everything and carry on...
        $wp_query->query_vars = $query_vars;
}

[phanduynam]

@RavanH I know I need to use an SEO plugin or a code replacement, but I don't know how to code and I want to use WordPress' built-in tools instead of a third-party plugin. Since this bug has been around for so long and is seriously affecting SEO, I have to fix it. I hope you will prioritize this issue and notify the WordPress core admins so they can include a fix in WordPress 6.8.3 or 6.9.

[RavanH]

Replying to phanduynam:

@RavanH I know I need to use an SEO plugin or a code replacement, but I don't know how to code and I want to use WordPress' built-in tools instead of a third-party plugin. Since this bug has been around for so long and is seriously affecting SEO, I have to fix it. I hope you will prioritize this issue and notify the WordPress core admins so they can include a fix in WordPress 6.8.3 or 6.9.

I agree, it needs a fix. The plugin I mentioned is light-weight and only adds some extra options that you do not need to use...

[phanduynam]

@RavanH The plugin you recommend hasn't been updated in a year, so I wouldn't use it.

[RavanH]

Replying to phanduynam:

@RavanH The plugin you recommend hasn't been updated in a year, so I wouldn't use it.

Updated :)

[phanduynam]

@RavanH Hi, I have translated to Vietnamese, if you are the Plugin creator, please approve this translation ​https://translate.wordpress.org/locale/vi/default/wp-plugins/xml-sitemaps-manager/

[RavanH]

Replying to phanduynam:

@RavanH Hi, I have translated to Vietnamese, if you are the Plugin creator, please approve this translation ​https://translate.wordpress.org/locale/vi/default/wp-plugins/xml-sitemaps-manager/

Thank you for the translation! I am not a Vietnamese translation editor but if you like, I can ask the team to make you a translation editor for the plugin?

[phanduynam]

@RavanH I remember the plugin creator has the right to approve translations, right?

[phanduynam]

@RavanH Why is there a change in the sitemap appearance? I want the sitemap to just fix the error and keep everything else the same to ensure its integrity.

[phanduynam]

@RavanH I think you should update the whole plugin, limit sitemap changes as much as possible and maybe refactor the plugin using AI.

[phanduynam]

maybe this bug will be fixed in wordpress 6.9

[jorbin]

I also tend to agree with @peterwilsoncc that #51117 is the root cause. I think we should close this as a duplicate.

[phanduynam]

@jorbin I think this is the original post, the error is still not fixed but is still marked as fixed https://core.trac.wordpress.org/ticket/50910