Quote:
Originally Posted by isf
It's BT who are making the google comparison and mentioning robots.txt as an opt-out for site operators, the user won't be requesting that file. As previously noted, not every site has access to the document root. If Phorm can append themselves to the user agent string, this gives web masters the ability to opt-out in a way that stands BT's comparison with googlebot.
|
I think it was also noted that they were doing a scrape for robots.txt of domains to keep in a cache, so during regular browsing of a Phorm connection, the kit still wont make its own unique connection to said site as it will already have the robots.txt on file, and thus leaving the connection purely down to the user agent of the user. There just wont be a user agent of Phorms to block.
---------- Post added at 17:58 ---------- Previous post was at 17:56 ----------
Rather than use the word 'block' I should have used the word 'deny' really as robots.txt is a system of honour and respecte rather than access protection.