Make WordPress Core

Opened 14 years ago

Closed 14 years ago

#11918 closed enhancement (fixed)

do_robots() Enhancement

Reported by: miqrogroove's profile miqrogroove Owned by:
Milestone: 3.0 Priority: normal
Severity: normal Version:
Component: Optimization Keywords:
Focuses: Cc:


do_robots(), in wp-includes/functions.php, is currently responsible for handling robots.txt requests for sites that do not have a robots.txt file.

The default rules are quite lame. All it does is allow or disallow the entire site based on the privacy setting.

There should be default rules such as

Disallow: /?s=
Disallow: /search/
Disallow: /wp-includes/
Disallow: *?replytocom

Attachments (1)

functions.php.diff (524 bytes) - added by secondv 14 years ago.

Download all attachments as: .zip

Change History (11)

#1 follow-up: @sirzooro
14 years ago

  • Cc sirzooro added

I do not think we should exclude search URLs here - for me this looks as a plugin area. Last 2 rules can be added. I would also add rule to exclude /wp-admin/. It will be also good to exclude /wp-content/ (at least plugin dir), but some plugins may provide files which should be indexed (although I do not know any at this moment), so this is not a good candidate.

#2 @secondv
14 years ago

  • Keywords has-patch added

I've attached a patch for this, against r12782

#3 @secondv
14 years ago

  • Cc secondv added

#4 in reply to: ↑ 1 @miqrogroove
14 years ago

-1 on secondv's patch. The lack of search rules makes this as useless as it was in the first place.

#5 @dd32
14 years ago

The lack of search rules makes this as useless as it was in the first place.

IMO, Theres no need to block search engines from Search results, It'll cause more load on some servers, but its up to the user to disable crawls of their search pages IMHO.

#6 @miqrogroove
14 years ago

Fair enough. I vote wontfix. Excluding wp-admin does very little in practice, and the absence of search exclusion is contrary to best practice.

#7 @sivel
14 years ago

  • Keywords has-patch removed
  • Milestone 3.0 deleted
  • Resolution set to wontfix
  • Status changed from new to closed

I agree with all of the -1 votes. Closing as wontfix.

#8 @nacin
14 years ago

See [13891] -- ryan has added a filter here.

I'm going to re-open this and close as fixed on that changeset.

#9 @nacin
14 years ago

  • Resolution wontfix deleted
  • Status changed from closed to reopened

#10 @nacin
14 years ago

  • Milestone set to 3.0
  • Resolution set to fixed
  • Status changed from reopened to closed

I know it's not what the reporter was looking for here, but a filter will allow implementation of all this.

Note: See TracTickets for help on using tickets.