built up sitemap

[omponk]

version 3.+ pls give a built up sitemap to search engine, like pinglist on options :)

[ramoonus]

this is plugin stuff
why tag is soo high

[ramoonus]

Replying to ramoonus:

this is plugin stuff
why tag is soo high

tag it ... typo

[jorbin]

There are already plugins that do this.

[offordscott]

sitemap.xml is a standard these days. I'd love to be able to generate one without installing a plugin to do it.

[ocean90]

Have in mind that you can use your feed as a sitemap too, at least Google and Bing supports it.

[offordscott]

Hi ocean90, that is interesting, but I don't think Pages are published in the feed, right? Only posts.

[igrigorik]

Hi folks.

I'd like to make the case that this ticket should be reopened and WordPress should provide a sitemap out of the box.

Why do sitemaps matter?

Sitemaps are a ​common protocol that enables crawlers to: efficiently discover and index content of a site, identify pages that it otherwise might miss, and quickly detect updates to previously indexed content. Instead of recrawling every page to find other pages, or detect if an update has been made, sitemaps enumerate the pages and their update timestamps. This helps speed up indexing and content discovery, which benefits greatly both the site owner as well as the crawler.

All popular search engines support and look for sitemaps when visiting a site (e.g. see ​Google docs), and having WordPress sites provide one by default would help the content get indexed and discovered faster.

What's the story today in WordPress?

(A) Site owner can install a plugin to generate a sitemap. This has multiple problems...

• The site owner needs to know that such a thing is needed. Most publishers won't, nor should they ever have to think about it. WordPress should come with all the batteries included to maximize my success on the web: having my content be discovered + indexed fast is critical.
• Because sitemaps are generated by 3P plugins today, there is no interop story between multiple plugins that may want to do this. For example, if SEO plugin A creates a sitemap, there is no well defined hook for SEO plugin B to detect this and avoid generating another variant that can cause both crawler and publisher confusion. We did an audit of popular plugins and none of them offer any form of detection / compatibility with other plugins. This can yield unpredictable results for the site owner.

(B) Some crawlers support RSS/Atom feeds in lieu of sitemap. However, this also has many problems...

• RSS feeds typically only enumerate a small (latest-N) number of pages, whereas the point of the sitemap is to provide the full list of content that should be crawled and indexed.
• RSS feeds generated by WP do not list all of the site’s content. For example, as raised earlier, pages are not listed in the RSS feed. In theory this could be, once again, fixed by a ​3P plugin, but now we're back to all the same problems as (A).
• If we did try to bend current RSS feeds to list all content, the resulting feed could be huge for some sites.
  • Sitemaps do not need the actual page content
  • Sitemaps can be split into multiple files

In short, neither of these solutions is good enough.

Wishlist for sitemap support in WordPress

WordPress should provide a sitemap file out of the box, with no action need by the site owner, same as the RSS feed:

1. WordPress core is responsible for generating standard XML sitemap.
2. Sitemaps should respect size limits ​enforced by popular search engines and split large sitemaps.
3. WordPress core should provide a standard API to enumerate available sitemaps, such that plugins can use this API to retrieve and submit these sitemaps on publishers behalf to relevant providers.
4. WordPress core should provide a notification hook for when a sitemap file is updated, to assist with (3). When a page, post, or other type of publicly accessible content is created or updated, an internal event should be fired.

[adamsilverstein]

Thanks for the feedback @igrigorik and re-opening for consideration.

There are already plugins that do this.

True. However if we accept the argument that (nearly) all websites should have a sitemap, maybe this is something that should be provided directly by core?

[johnbillion]

I'm inclined to agree that this is a feature that serves the majority of WordPress users (and therefore fits the 80/20 guideline). That said, such a feature would need to be maintained otherwise it risks going stale and not being as beneficial as a regularly updated third party plugin.

I wonder what are the thoughts from @yoast?

[joostdevalk]

We strongly agree that this should be in core. It's something we've put forth as a feature project before, which was accepted at that point but then sidelined because of Gutenberg. I've discussed it with @matt before as well because Jetpack and WP.com have an XML sitemap implementation too.

We are willing to do some architectural work for this, to lay out what we think would be a good way of generating these in WordPress.

[adamsilverstein]

We are willing to do some architectural work for this, to lay out what we think would be a good way of generating these in WordPress.

Thank you for the fantastic offer, your expertise in building sitemaps will be incredibly valuable.

Jetpack and WP.com have an XML sitemap implementation too

I will review the Jetpack implementation, we have often started building core features by leveraging established open source solutions.

[swissspidy]

Work on this is under way...

​https://make.wordpress.org/core/2020/06/10/merge-announcement-extensible-core-sitemaps/

See #50117