Do we actually know if it is Google they look for in robots.txt or is it the any agent string? I posed this question a bit back and (unless I missed it) there was no answer:
Quote:
Originally Posted by jelv
Does anyone know if they are using the specific user agent for Google or is it the any agent setting they use? So would
Code:
User-agent: Google
Disallow:
User-agent: *
Disallow: /
work as it should - Google allowed in but Phorm/Webwise kept out? |
If (unlikely as it is) they do obey the robots.txt rules we need a robots.txt file putting together which includes all the known valid agents and barring *