Native UI to Manage robots.txt Content from WordPress Admin

[samahjoob12]

WordPress automatically generates a virtual robots.txt file if one does not physically exist. However, there is no built-in interface to manage or customize its contents.

This enhancement proposes a simple native UI in the WordPress admin (e.g., under Settings > Reading) to allow users to view and optionally override the default virtual robots.txt output.

Proposed Behavior:

If no physical robots.txt exists, WordPress will continue to serve a virtual version — but users can edit its content via the new UI.

If a physical robots.txt is found in the root directory, display a notice and provide options:

View current content (read-only)

Delete and switch to virtual + editable version

Optional: basic syntax check or linter to prevent common formatting errors

Benefits:

Improves UX for users who want better SEO control without using plugins or FTP.

Encourages safer and more transparent management of crawler rules.

Bridges a long-standing gap between virtual behavior and user expectations.

Makes the default robots.txt output editable — a much-requested feature.

Why this belongs in Core:

It's a light, low-risk enhancement that doesn’t interfere with existing plugin functionality.

Fits the WordPress philosophy of empowering users while keeping things simple.

Many popular plugins (like Yoast, RankMath) already implement this — core should offer a basic version.

[audrasjb]

Hello and thanks for the feature proposal,

I think we're clearly in the plugin territory as WordPress automatically handles the robots.txt and also provides several filters and action for plugin developers to extend the default behavior. Our "Decision not Option" guidelines states that a feature should be useful for a clear majority of users, otherwise it belongs to plugin extenders. I feel also a bit concerned about how complex it would be for the average user, who would probably prefer for this settings to be managed automatically by the CMS.

I'd probably close this proposal as wontfix or maybelater, but leaving it open for now for further feedback.

[karmatosed]

I agree with the recommendation to close. This to me also feels like plugin territory so for now I am going to move this through to close with wontfix. Thank you, we can always reconsider should further information come up.