In a WebmasterWorld thread many Webmasters are reporting heavy Googlebot activity. One member's Server was inadvertantly shut down with a Denial of Service (DoS).
One member had to resort to temporarily banning the rampant Googlebot IP address (69.249.66.51) off and on to control the indexing of his site. Googlebot 2.1, the new bot in town, had been requesting 20 pages per second says this member.
GoogleGuy responded with "the crawl team is looking into it. We don't want to crawl so hard that you have to take action like that."
I agree, this is a little overboard and out of control. Perhaps what Google needs to do is institute the same robots.txt directive as MSN and Yahoo have -- the Crawl-Delay -- which will allow individual site owners to have control over how fast their sites are crawled.
The crawl-delay is easy enough to use. With three major players in the arena now, and all thirsty to keep their indexes fresh, some type of regulation is definitely needed.
No comments:
Post a Comment