Make WordPress Core

Opened 20 months ago

Last modified 2 months ago

#56595 new enhancement

Add a Site Health check for a non-virtual robots.txt file

Reported by: zodiac1978's profile zodiac1978 Owned by:
Milestone: Awaiting Review Priority: normal
Severity: normal Version:
Component: Site Health Keywords: dev-feedback has-patch
Focuses: Cc:

Description

At WordCamp Nederland 2022 @joostdevalk held a talk about unnecessary bot traffic and how to prevent it.

One slide caught my interest:
https://docs.google.com/presentation/d/13Ngq-T2Qdbz1b8apUiioTCBmcsB5s411xBKcklmKyNQ/edit#slide=id.g152f65bfa26_0_87

Blocking those unneeded bots is easy in theory (there is a filter available to change the virtual robots.txt file), but is not easy to build, because we need to look at many use cases and edge cases.

For high traffic sites, it would be better to have a non-virtual robots.txt file, to prevent PHP/WordPress handling this.

But if we create a robots.txt file it is easily missed that now WordPress is not handling it anymore.

Therefore I suggest adding a check to Site Health if there is non-virtual robots.txt file in the root directory.

Maybe we could also add the content of this file in the info area and/or in the tools section of the plugin.

Happy to work on a patch if this idea gets confirmation.

Attachments (4)

56595.diff (1.2 KB) - added by zodiac1978 2 months ago.
Adding a robots.txt check to the debug data in the Site Health feature
virtual-robotstxt.png (14.8 KB) - added by zodiac1978 2 months ago.
Debug info if there is no pyhsical robots.txt file available
physical-robotstxt.png (15.8 KB) - added by zodiac1978 2 months ago.
Debug info if there is a pyhsical robots.txt file available
debug-info.png (11.2 KB) - added by zodiac1978 2 months ago.
Debug info as shown in the clipboard data

Download all attachments as: .zip

Change History (6)

#1 @zodiac1978
20 months ago

  • Type changed from defect (bug) to enhancement

#2 @zodiac1978
2 months ago

  • Keywords has-patch added; needs-patch removed

Maybe we could also add the content of this file in the info area and/or in the tools section of the plugin.

Since version 1.7.0 there is a robots.txt viewer in the Health Check plugin available:
https://github.com/WordPress/health-check/commit/015347bc7cf1b5fb81281add0a7db5d4f7b5de66

Now I would like to add the according debug data.

@zodiac1978
2 months ago

Adding a robots.txt check to the debug data in the Site Health feature

@zodiac1978
2 months ago

Debug info if there is no pyhsical robots.txt file available

@zodiac1978
2 months ago

Debug info if there is a pyhsical robots.txt file available

@zodiac1978
2 months ago

Debug info as shown in the clipboard data

Note: See TracTickets for help on using tickets.