Blog Documentation Book a Demo
Vanilla 1 is no longer supported or maintained. If you need a copy, you can get it here.
HackerOne users: Testing against this community violates our program's Terms of Service and will result in your bounty being denied.

Google indexing question: goog stopped indexing the site

crash_D.crash_D.
edited June 2007 in Vanilla 1.0 Help
Hi all, one of the things I do is SEO, so i am not a complete amateur in it, but I see such a phenomenon for the first time... I have seen goog excluding my site for keyword spamming and other black hat SEO tricks, I have seen goog praise sites with no valid reason, I have seen goog never index a certain site, etc. etc. BUT. this time, google has simply stopped indexing a site without any reason known to me... he accessed the homepage on march 18 and then never came back :( ... we did not include any meta tags that influence indexing (which in fact they don't).. we also don't have any robots.txt file to prevent indexing...

In addition, i have installed that auto site to my goog webmaster tools a while ago, and starting from march, goog hasnt been able to verify ownership (or administration in my case).. I have uploaded the verification file as it demanded, but still googbot could not access it; and even more, goog did not give a meaningful explanation for this (ok, maybe im too hard on him)... so, as the screenshot (54kb) illustrates, it has a temporary problem ....

the question is very simple: any ideas?

thanks... any thoughts would be appreciated, couse this is the first time i see such a strange thing... thanks

Comments

  • WallPhoneWallPhone
    When googlebot says this:GET /robots.txt HTTP/1.1 Host: autoplus.com.ua Connection: close
    Your server says this:HTTP/1.1 500 Internal Server Error ... <title>Invalid attempt to read when no data is present.</title> ...
    Googlebot probably thinks your server is down and doesn't bother continuing after that it sees that message.

    Fix? configure the server to return a proper 404, or upload a blank/permissive robots.txt
  • crash_D.crash_D.
    ok Wallphone, you just showed me I AM an amateur... where did you receive this data??
  • crash_D.crash_D.
    you are the best... I remember that I have setup custom error pages... once I have reset the custom page to default 404 page, the problem vanished and the webmaster tools verified the site, and I hope the bot would now save the current copy... thanks WallPhone..!!!
  • crash_D.crash_D.
    lol... i have asked the hosting company about this problem before i turned to lussumo, and this is what i received this morning: secondly you have asked about the google that it cant index the site basically
    Google servers are experiencing some problems especially GoogleBOT,
    it will be good soon.
  • WallPhoneWallPhone
    edited June 2007
    I kinda stumbled onto that.

    My first suggestion was to check the server log files to see what they say the last visits of Googlebot were, and then to check the status codes to make sure it wasn't getting 403's or 500's which litterally mean 'go away, your not welcome' or 'i'm sick at the moment, come back later''.

    Then I figured I could see the statuscodes for myself, and went to Rex Swain's nifty HTTP viewer and punched in the path to your robots.txt.

    Google really should tell you a better error message there. Probably a good thing to send to Matt Cutts.
  • dan39dan39 New
    edited June 2007
    I'm pretty sure the Google Webmaster Tools already tell you when the Error codes are wrong. I know I've seen that error before myself.

    BTW, I'm also pretty sure Matt Cutts is not the best person to send that to. As far as I know, Matt Cutts develops anti-spam algorithms. I don't think he works on the Webmaster Tools application.

    I'd send your comments in using the "Send Google Feedback" form instead.
  • idonnyidonny
    Have you considered the possibility that if your site is hosted o a shared-hosting server, you may be suffering from the reputation of another site hosted on the same IP address that has been blacklisted by google. If a site on the same IP address as yours gets banned by google, there is a very likely possibility that the IP address is banned too and even if our domain name may not be blacklisted, google will not index all domains hosted on the baned IP address
This discussion has been closed.