I've had some new replies from BT - paraphrasing, here goes :
It's use of the robots.txt is to determine whether or not the page should be deemed to be in the "public domain"
Quote:
|
it is assumed that the page is available to Webwise as long as the requester of the page consents, and we are not informed otherwise by the Website.
|
But then it explains that it doesn't crawl like a bot, but uses the information in robots.txt to decide whether or not the page contents should be analyzed.
Quote:
|
The system uses the rules set out in the robots.txt file which apply to the Googlebot user-agent.
|
It then goes on to explain that it is the responsibility of the broadband subscriber to explain to casual users of their systems (family, friends, etc) that Webwise is turned on - they liken it to security settings.
All in all with another couple of bits they have really made me quite angry again.
I am trying to see how robots.txt can stop access to password protected pages, but I didn't ask that and I don't have a reference to a quote about that - has anyone got one knocking around?