Make WordPress Core

Opened 10 years ago

Closed 10 years ago

#28556 closed defect (bug) (invalid)

robots.txt is not auto generated.

Reported by: nicolapeluchetti's profile nicola.peluchetti Owned by:
Milestone: Priority: normal
Severity: normal Version: 3.9.1
Component: Query Keywords: close
Focuses: Cc:

Description

If i have no robots.txt in my wordpress install and visit the robots.txt file in the browser, nothing is generated and a 404 is returned. No plugin installed, no permalinks enabled.
I've looked at the code and there doesn't seem to be any check for crawler user agent and i've got an old site in wordpress 3.5.1 which actually works.
I was trying to test a function that adds my rules to robots.txt.
To reproduce, simply try to visit the robots.txt file in your browser.
If i force it in query.php by returning true from is_robots, also a notice appears.

Attachments (3)

robots.png (35.4 KB) - added by nicola.peluchetti 10 years ago.
404
robots working.png (52.6 KB) - added by nicola.peluchetti 10 years ago.
working
notice.png (53.3 KB) - added by nicola.peluchetti 10 years ago.
notice when forced

Download all attachments as: .zip

Change History (7)

@nicola.peluchetti
10 years ago

404

@nicola.peluchetti
10 years ago

working

@nicola.peluchetti
10 years ago

notice when forced

#1 @nicola.peluchetti
10 years ago

Sorry i meant "Pretty permalinks" enabled, don't know how to change the body of the ticket-

#2 @SergeyBiryukov
10 years ago

  • Keywords close added

WordPress only responds to a robots.txt request when installed at the root of a domain:
tags/3.9.1/src/wp-includes/rewrite.php#L1551.

#4 @dd32
10 years ago

  • Milestone Awaiting Review deleted
  • Resolution set to invalid
  • Status changed from new to closed

robots.txt is only valid at the root of a domain, when WordPress is installed in a sub directory (and not using a root domain as the home_url) then robots.txt shouldn't be served.

Note: See TracTickets for help on using tickets.