do_robots() Enhancement

[miqrogroove]

do_robots(), in wp-includes/functions.php, is currently responsible for handling robots.txt requests for sites that do not have a robots.txt file.

The default rules are quite lame. All it does is allow or disallow the entire site based on the privacy setting.

There should be default rules such as

Disallow: /?s=
Disallow: /search/
Disallow: /wp-includes/
Disallow: *?replytocom

[sirzooro]

I do not think we should exclude search URLs here - for me this looks as a plugin area. Last 2 rules can be added. I would also add rule to exclude /wp-admin/. It will be also good to exclude /wp-content/ (at least plugin dir), but some plugins may provide files which should be indexed (although I do not know any at this moment), so this is not a good candidate.

[secondv]

I've attached a patch for this, against r12782

[miqrogroove]

-1 on secondv's patch. The lack of search rules makes this as useless as it was in the first place.

[dd32]

The lack of search rules makes this as useless as it was in the first place.

IMO, Theres no need to block search engines from Search results, It'll cause more load on some servers, but its up to the user to disable crawls of their search pages IMHO.

[miqrogroove]

Fair enough. I vote wontfix. Excluding wp-admin does very little in practice, and the absence of search exclusion is contrary to best practice.

[sivel]

I agree with all of the -1 votes. Closing as wontfix.

[nacin]

See [13891] -- ryan has added a filter here.

I'm going to re-open this and close as fixed on that changeset.

[nacin]

I know it's not what the reporter was looking for here, but a filter will allow implementation of all this.