Adding a robot.txt file to WordPress

[hardeepasrani]

I don't think I'm the only one with this argument, and I won't be surprised if you guys mark this ticket as duplicated, but still let's talk about it.

WordPress comes out with a .htaccess file and you can see the file in the root, but you now we add a robot.txt file to WordPress, which works just like .htaccess concept.

WordPress can modify that file, and it will also be available on the root, so a user can edit it as well. What do you think about this, duplicate or anything?

[SergeyBiryukov]

Previously: #1949

We do have a function (added in [3791]) that responds to a robots.txt request if WP is installed at the root of a domain: tags/3.8.3/src/wp-includes/functions.php#L1085​.

[hardeepasrani]

Looks good from the change set, but there should be a .txt file in the root as well, if anyone can create that. I guess the main problem would be that we can't limit the robot.txt file, so that it works same as the WordPress.

Shall we mark this as wontfix?

[SergeyBiryukov]

Anyone can create a robots.txt file in the root directory, and it will take precedence.

I don't see a reason to bundle the file with WordPress.