Do not add trailing slash to the robots.txt URL
|Reported by:||thenlich||Owned by:||anonymous|
The /robots.txt is redirected to /robots.txt/ if configured. This is in violation of the robot exclusion standard, which specifies
This file must be accessible via HTTP on the local URL "/robots.txt".
It is also not network nice because it generates extra traffic for the additional request after redirection.