Robots.txt exclusion rule order reversal

[rdela]

This is a follow-up to #33156.

Replying to comment:17 Hube2:

Didn't know if I should start a new ticket or not and couldn't find one covers it. The order that WP is outputting the allow/disallow rules

Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

may not compatible with all crawlers. To be comparable with all crawlers the order of these rules needs to be reversed.

See: ​Wikipedia – Robots exclusion standard: Allow directive

I can't find any information that contradicts what is presented in the wiki article.

​Google’s example is in Allow then Disallow order.

Pattern-matching rules to streamline your robots.txt code

User-agent: *
Allow: /*?$
Disallow: /*?

Google's ​robots.txt Tester
confirms both orders work for googlebot.

[dmchale]

According to ​Goolge's robots.txt specifications page ... (emphasis mine)

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

Therefore Allow: /wp-admin/admin-ajax.php will trump Disallow: /wp-admin/ no matter which order they are listed

Personally, before making any changes I would also like to see other evidence which corroborates the ​Wikipedia article's assertion that by standard implementation the first matching robots.txt pattern always wins. There is no citation on that claim and I'd love to see something more concrete than one editor's choice of phrasing.

[SergeyBiryukov]

Per comment:1, the existing rules appear to work as expected. Feel free to reopen with additional information.