Quote:
Originally Posted by jelv
Do we actually know if it is Google they look for in robots.txt or is it the any agent string? I posed this question a bit back and (unless I missed it) there was no answer:
|
I have seen nothing specific about this - they have kept that deliberately vague, its one of the things they don't want to talk about.
Reading between the lines, I think that what they are looking for is a straightforward robots.txt ban on the page or folder or whole site, or a noindex,nofollow metatag.
But - they are not coming clean on the details, so once again - no proper inphormation available for inphormed consent for customer, let alone webmasters.
You can check out the BT Webwise FAQ here without putting yourself at risk of picking up any webwise.bt.com cookies - I got BT to create a mirror site for the FAQ on their own servers (originally all the Webwise stuff was only available on rather flakey FASTHOSTS servers, outside the BT network
It contains the information that BT are prepared to put record. You'll notice how good they are at not really providing the information we really need to know.
It also tends to be out of date - but then that can work in our favour because you can quote that information to management and ask them if it is still their official line - at which point they sometimes have to go away and have a little think - and the trials go back another month!
http://www.productsandservices.bt.co...=CON-WEBWISE-I