id,summary,reporter,owner,description,type,status,priority,milestone,component,version,severity,resolution,keywords,cc
23070,Generated robots.txt file with privacy on should have Allow: /robots.txt,iamfriendly,,"Scenario:

You're developing a site online (perhaps a new version of an existing site) and you have privacy settings disallowing (or, rather, discouraging) crawling. This generates a robots.txt file with


{{{
User-agent: *
Disallow: /
}}}


Now, you've finished the new site and want google to have a look, so you stop discouraging spiders. This can take up to 48 hours for google to respider your robots.txt file. There's a way 'around' that by using Google Webmaster Tools and using the 'Fetch as Google' tool (under 'Health').

If the generated 'private' robots.txt file had


{{{
Allow: /robots.txt
}}}

in it, then you would be able to force google to respider your robots.txt file (and hence allow it to spider your sitemap.xml file) when you need it to, rather than having to wait for an indeterminate amount of time.

Granted, there are ways in which you can avoid this scenario, but I also know there are circumstances where you can't, and adding that simple one-liner can help you out.

Am I missing a trick? Am I being stupid? If not, then I'll write the very quick patch.",enhancement,closed,normal,,General,,normal,invalid,,