WordPress.org

Make WordPress Core

#35819 closed enhancement (invalid)

Robots.txt exclusion rule order reversal

Reported by: rdela Owned by:
Milestone: Priority: normal
Severity: normal Version: 4.4.2
Component: General Keywords:
Focuses: administration Cc:

Description

This is a follow-up to #33156.

Replying to comment:17 Hube2:

Didn't know if I should start a new ticket or not and couldn't find one covers it. The order that WP is outputting the allow/disallow rules

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

may not compatible with all crawlers. To be comparable with all crawlers the order of these rules needs to be reversed.

See: Wikipedia – Robots exclusion standard: Allow directive

I can't find any information that contradicts what is presented in the wiki article.

Google’s example is in Allow then Disallow order.

Pattern-matching rules to streamline your robots.txt code

User-agent: *
Allow: /*?$
Disallow: /*?

Google's robots.txt Tester
confirms both orders work for googlebot.

Change History (2)

#1 @dmchale
22 months ago

According to Goolge's robots.txt specifications page ... (emphasis mine)

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

Therefore Allow: /wp-admin/admin-ajax.php will trump Disallow: /wp-admin/ no matter which order they are listed

Personally, before making any changes I would also like to see other evidence which corroborates the Wikipedia article's assertion that by standard implementation the first matching robots.txt pattern always wins. There is no citation on that claim and I'd love to see something more concrete than one editor's choice of phrasing.

Last edited 22 months ago by dmchale (previous) (diff)

#2 @SergeyBiryukov
12 months ago

  • Milestone Awaiting Review deleted
  • Resolution set to invalid
  • Status changed from new to closed

Per comment:1, the existing rules appear to work as expected. Feel free to reopen with additional information.

Note: See TracTickets for help on using tickets.