Block The Baidu Spider From My Site?

Callumpy · 08-06-2009, 22:52

Hello... does anyone know how i can block the Baidu spider from visiting my website? Ive tried a robots.txt but that hasnt seemed to have worked.

Acathla · 08-06-2009, 23:26

You can do this but it will still make server requests - just not index anything:

Code:

SetEnvIfNoCase User-Agent "^Baidu" bad_bot
<Directory />
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</Directory>

You could probably also see about doing a mod_rewrite like:

Code:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^baiduspider [NC]
RewriteRule .* - [F]

**Paul** · 09-06-2009, 01:55

Why do you want to block it ?

Callumpy · 09-06-2009, 17:23

I want to block it because my site is on my home server and it visits way to often. Eating my bandwidth.

Acathla, where do I put those?

Dai · 09-06-2009, 19:03

Quote:

Originally Posted by Callumpy

I want to block it because my site is on my home server and it visits way to often. Eating my bandwidth.

Acathla, where do I put those?

Apparently it also ignores the robots.txt file so you may find it indexing bits you would rather keep private.

I'm not sure why anyone would want to be on a Chinese search engine anyway?

Callumpy · 09-06-2009, 22:27

Ive realised it ignores the robots.txt...

User-agent: BaiDuSpider
Disallow: /

User-agent: Baidu
Disallow: /

User-agent: Baiduspider+(+http://www.baidu.com/search/spider.htm)
Disallow: /

User-agent: Baiduspider+(+http://www.baidu.com/search/spider_jp.html)
Disallow: /

The only annoying thing is that it visits like every 5 - 10 mins. Its eating my bandwidth.
I cant block it by IP either cause it has loads of IPs

Acathla · 09-06-2009, 22:45

OK I guess I should really have asked how your hosting / server / etc as this is for Apache.

Into your httpd.conf

Let me know what you can configure and I'll see if I can help.

Callumpy · 10-06-2009, 19:02

I use abyss

Dai · 10-06-2009, 20:38

Quote:

Originally Posted by Callumpy

The only annoying thing is that it visits like every 5 - 10 mins. Its eating my bandwidth.
I cant block it by IP either cause it has loads of IPs

Block the whole 60.28.*.* range.
Unless you have some reason to want Chinese visitors that'll keep a lot of the spiders out.

AbyssUnderground · 11-06-2009, 14:14

I can't see how 1 spider can eat all your bandwidth, I used to host my site on half the bandwidth you have and I had google spider and loads of other spiders crawling my site all the time, and it being a forum it had tens of thousands of pages to crawl. It used barely anything, its just like a normal visitor.

The only reason it might use lots is if it has no request-delay like the other crawlers do. Google's default is 5 seconds/page so if its doing more than 1/second it might use more.

As said above the easiest way to block it if it is ignoring the robots file is to block the IP range.