WordPress.org

Make WordPress Core

Opened 4 years ago

Closed 4 years ago

#11918 closed enhancement (fixed)

do_robots() Enhancement

Reported by: miqrogroove Owned by:
Milestone: 3.0 Priority: normal
Severity: normal Version:
Component: Optimization Keywords:
Focuses: Cc:

Description

do_robots(), in wp-includes/functions.php, is currently responsible for handling robots.txt requests for sites that do not have a robots.txt file.

The default rules are quite lame. All it does is allow or disallow the entire site based on the privacy setting.

There should be default rules such as

Disallow: /?s=
Disallow: /search/
Disallow: /wp-includes/
Disallow: *?replytocom

Attachments (1)

functions.php.diff (524 bytes) - added by secondv 4 years ago.

Download all attachments as: .zip

Change History (11)

comment:1 follow-up: sirzooro4 years ago

  • Cc sirzooro added

I do not think we should exclude search URLs here - for me this looks as a plugin area. Last 2 rules can be added. I would also add rule to exclude /wp-admin/. It will be also good to exclude /wp-content/ (at least plugin dir), but some plugins may provide files which should be indexed (although I do not know any at this moment), so this is not a good candidate.

comment:2 secondv4 years ago

  • Keywords has-patch added

I've attached a patch for this, against r12782

secondv4 years ago

comment:3 secondv4 years ago

  • Cc secondv added

comment:4 in reply to: ↑ 1 miqrogroove4 years ago

-1 on secondv's patch. The lack of search rules makes this as useless as it was in the first place.

comment:5 dd324 years ago

The lack of search rules makes this as useless as it was in the first place.

IMO, Theres no need to block search engines from Search results, It'll cause more load on some servers, but its up to the user to disable crawls of their search pages IMHO.

comment:6 miqrogroove4 years ago

Fair enough. I vote wontfix. Excluding wp-admin does very little in practice, and the absence of search exclusion is contrary to best practice.

comment:7 sivel4 years ago

  • Keywords has-patch removed
  • Milestone 3.0 deleted
  • Resolution set to wontfix
  • Status changed from new to closed

I agree with all of the -1 votes. Closing as wontfix.

comment:8 nacin4 years ago

See [13891] -- ryan has added a filter here.

I'm going to re-open this and close as fixed on that changeset.

comment:9 nacin4 years ago

  • Resolution wontfix deleted
  • Status changed from closed to reopened

comment:10 nacin4 years ago

  • Milestone set to 3.0
  • Resolution set to fixed
  • Status changed from reopened to closed

I know it's not what the reporter was looking for here, but a filter will allow implementation of all this.

Note: See TracTickets for help on using tickets.