HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.
robots.txt on Vanilla clients forums
phreak
MVP
Hi Vanilla's,
I found that some of the clients forums of Vanilla do have this Disallows in the robots.txt.
User-agent: * Disallow: /entry/ Disallow: /messages/ Disallow: /profile/comments/ Disallow: /profile/discussions/ Disallow: /search/
Can anyone tell me why to disallow the comments and the discussions through the profile URL? Aren't they indexed already? Wouldn't it be smarter to disallow the whole /profile/ directory.
Just asking. @Adrian, do you know more about it.
Thanx, phreak
- VanillaAPP | iOS & Android App for Vanilla - White label app for Vanilla Forums OS
- VanillaSkins | Plugins, Themes, Graphics and Custom Development for Vanilla
Tagged:
0
Comments
@phreak, The only thought I can have is this comment from Todd . Frankly I've never see any value in profile pages being indexed at all, so I normally don't bother in my self-hosted installs. I've never seen any issue from an SEO perspective in doing this in any way.
Disallow: /profile/Thanks @Adrian. I will make 2 test cases on a 100.000 UC website and report here. I hope it's only going up.
Our concern was that it would be duplicate content to have posts/lists on the profile.
Ok, could you ever verify this or was it solved through proper implementation of the canonical tag and is some kind of leftover from that time (i posted the robots.txt question 2010).
No, I don't think anyone has looked at it further. My inclination is that it seems like a correct decision and I don't see value in reversing it.