robots.txt on Vanilla clients forums

phreakphreak VanillaAPP - White label iOS and Android App MVP
edited February 2016 in General Banter

Hi Vanilla's,

I found that some of the clients forums of Vanilla do have this Disallows in the robots.txt.

User-agent: * Disallow: /entry/ Disallow: /messages/ Disallow: /profile/comments/ Disallow: /profile/discussions/ Disallow: /search/

Can anyone tell me why to disallow the comments and the discussions through the profile URL? Aren't they indexed already? Wouldn't it be smarter to disallow the whole /profile/ directory.

Just asking. @Adrian, do you know more about it.

Thanx, phreak


  • Vanilla APP » Learn more «
  • iOS & Android App for Vanilla - White label app for your forum
Tagged:

Comments

  • AdrianAdrian Wandering Spirit Montreal Vanilla Staff

    @phreak, The only thought I can have is this comment from Todd . Frankly I've never see any value in profile pages being indexed at all, so I normally don't bother in my self-hosted installs. I've never seen any issue from an SEO perspective in doing this in any way.

    Disallow: /profile/

    Sharing is caring

  • phreakphreak VanillaAPP - White label iOS and Android App MVP

    Thanks @Adrian. I will make 2 test cases on a 100.000 UC website and report here. I hope it's only going up. ;) o:)


    • Vanilla APP » Learn more «
    • iOS & Android App for Vanilla - White label app for your forum
  • LincLinc Director of Development Detroit Vanilla Staff

    Our concern was that it would be duplicate content to have posts/lists on the profile.

    Adrian
  • phreakphreak VanillaAPP - White label iOS and Android App MVP

    Ok, could you ever verify this or was it solved through proper implementation of the canonical tag and is some kind of leftover from that time (i posted the robots.txt question 2010).


    • Vanilla APP » Learn more «
    • iOS & Android App for Vanilla - White label app for your forum
  • LincLinc Director of Development Detroit Vanilla Staff

    @phreak said:
    Ok, could you ever verify this or was it solved through proper implementation of the canonical tag and is some kind of leftover from that time (i posted the robots.txt question 2010).

    No, I don't think anyone has looked at it further. My inclination is that it seems like a correct decision and I don't see value in reversing it.

Sign In or Register to comment.