Thread: Block The Baidu Spider From My Site?
View Single Post
Old 11-06-2009, 14:14   #10
AbyssUnderground
Inactive
 
Join Date: Oct 2005
Location: Merseyside
Age: 37
Services: BT Infinity Option 2, HH5, synced at maximum 80Mbps/20Mbps.
Posts: 2,221
AbyssUnderground has reached the bronze age
AbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze ageAbyssUnderground has reached the bronze age
Send a message via MSN to AbyssUnderground
Re: Block The Baidu Spider From My Site?
I can't see how 1 spider can eat all your bandwidth, I used to host my site on half the bandwidth you have and I had google spider and loads of other spiders crawling my site all the time, and it being a forum it had tens of thousands of pages to crawl. It used barely anything, its just like a normal visitor.

The only reason it might use lots is if it has no request-delay like the other crawlers do. Google's default is 5 seconds/page so if its doing more than 1/second it might use more.

As said above the easiest way to block it if it is ignoring the robots file is to block the IP range.
AbyssUnderground is offline   Reply With Quote