robots.txt on Vanilla clients forums
I found that some of the clients forums of Vanilla do have this Disallows in the robots.txt.
User-agent: * Disallow: /entry/ Disallow: /messages/ Disallow: /profile/comments/ Disallow: /profile/discussions/ Disallow: /search/
Can anyone tell me why to disallow the comments and the discussions through the profile URL? Aren't they indexed already? Wouldn't it be smarter to disallow the whole /profile/ directory.
Just asking. @Adrian, do you know more about it.