Do not add trailing slash to the robots.txt URL

[thenlich]

The /robots.txt is redirected to /robots.txt/ if configured. This is in violation of the robot exclusion standard, which specifies

This file must be accessible via HTTP on the local URL "/robots.txt".

It is also not network nice because it generates extra traffic for the additional request after redirection.

[DD32]

I spotted this the other day, Didnt get around to submitting a trac ticket though, Thanks for bringing it up and reminding me :)

[ryan]

(In [7871]) Don't redirect robots. Props thenlich and DD32. fixes #6891