WordPress.org

Make WordPress Core

Opened 5 years ago

Closed 4 years ago

#28556 closed defect (bug) (invalid)

robots.txt is not auto generated.

Reported by: nicola.peluchetti Owned by:
Milestone: Priority: normal
Severity: normal Version: 3.9.1
Component: Query Keywords: close
Focuses: Cc:

Description

If i have no robots.txt in my wordpress install and visit the robots.txt file in the browser, nothing is generated and a 404 is returned. No plugin installed, no permalinks enabled.
I've looked at the code and there doesn't seem to be any check for crawler user agent and i've got an old site in wordpress 3.5.1 which actually works.
I was trying to test a function that adds my rules to robots.txt.
To reproduce, simply try to visit the robots.txt file in your browser.
If i force it in query.php by returning true from is_robots, also a notice appears.

Attachments (3)

robots.png (35.4 KB) - added by nicola.peluchetti 5 years ago.
404
robots working.png (52.6 KB) - added by nicola.peluchetti 5 years ago.
working
notice.png (53.3 KB) - added by nicola.peluchetti 5 years ago.
notice when forced

Download all attachments as: .zip

Change History (7)

@nicola.peluchetti
5 years ago

404

@nicola.peluchetti
5 years ago

working

@nicola.peluchetti
5 years ago

notice when forced

#1 @nicola.peluchetti
5 years ago

Sorry i meant "Pretty permalinks" enabled, don't know how to change the body of the ticket-

#2 @SergeyBiryukov
5 years ago

  • Keywords close added

WordPress only responds to a robots.txt request when installed at the root of a domain:
tags/3.9.1/src/wp-includes/rewrite.php#L1551.

#4 @dd32
4 years ago

  • Milestone Awaiting Review deleted
  • Resolution set to invalid
  • Status changed from new to closed

robots.txt is only valid at the root of a domain, when WordPress is installed in a sub directory (and not using a root domain as the home_url) then robots.txt shouldn't be served.

Note: See TracTickets for help on using tickets.