Add a Site Health check for a non-virtual robots.txt file

[zodiac1978]

At WordCamp Nederland 2022 @joostdevalk held a talk about unnecessary bot traffic and how to prevent it.

One slide caught my interest:
​https://docs.google.com/presentation/d/13Ngq-T2Qdbz1b8apUiioTCBmcsB5s411xBKcklmKyNQ/edit#slide=id.g152f65bfa26_0_87

Blocking those unneeded bots is easy in theory (there is a filter available to change the virtual robots.txt file), but is not easy to build, because we need to look at many use cases and edge cases.

For high traffic sites, it would be better to have a non-virtual robots.txt file, to prevent PHP/WordPress handling this.

But if we create a robots.txt file it is easily missed that now WordPress is not handling it anymore.

Therefore I suggest adding a check to Site Health if there is non-virtual robots.txt file in the root directory.

Maybe we could also add the content of this file in the info area and/or in the tools section of the plugin.

Happy to work on a patch if this idea gets confirmation.

[zodiac1978]

Maybe we could also add the content of this file in the info area and/or in the tools section of the plugin.

Since version 1.7.0 there is a robots.txt viewer in the Health Check plugin available:
​https://github.com/WordPress/health-check/commit/015347bc7cf1b5fb81281add0a7db5d4f7b5de66

Now I would like to add the according debug data.

[zodiac1978]

Adding a robots.txt check to the debug data in the Site Health feature

[zodiac1978]

Debug info if there is no pyhsical robots.txt file available

[zodiac1978]

Debug info if there is a pyhsical robots.txt file available

[zodiac1978]

Debug info as shown in the clipboard data

[audrasjb]

Moving for 6.8 consideration

https://core.trac.wordpress.org/ticket/56595

[audrasjb]

I added a PR with your patch as it didn't apply anymore against trunk @zodiac1978 :)

[zodiac1978]

Thanks @audrasjb !

I was just on my way to look if a refresh is necessary. You beat me to it. Thanks again!

[zodiac1978]

As the link to the slide is broken, here it is as image

[jorbin]

I'm not sure physical is the best way to refer to a digital file existing otherwise this looks good to me. How about changing it to be static for the files that live on the file system and and dynamic for the WordPress generated ones?

[audrasjb]

👆 The above wording proposal makes a lot of sense to me.

[jorbin]

I've updated the PR and intend to commit it tomorrow if there is no objection. As a part of this, I also added a check to make sure that rewrite rules allow before saying that WP can serve the dynamic file.

[jorbin]

In 59890:

Site Health: Add a robots.txt check to the server data.

Provide a bit of information about robots.txt to help people understand if the file is generated by WordPress.

Props zodiac1978, audrasjb, joostdevalk, jorbin.
Fixes #56595.

[SergeyBiryukov]

In 59894:

Site Health: Fix typo in the robots.txt check messages.

Follow-up to [59890].

See #56595.