|
Re: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
Quote:
Originally Posted by R Jones
<snip>as far as google is concerned the only robots.txt it sees is the one at the top level domain
|
Yup. Google is just following the RFC for robots.txt here. It should only look in the root of each domain. However Google also checks for meta tags on each page. This is one reason why checking robots.txt isn't good enough.
http://www.robotstxt.org/faq/shared.html
|