Quote:
Originally Posted by JackSon
I think it was also noted that they were doing a scrape for robots.txt of domains to keep in a cache, so during regular browsing of a Phorm connection, the kit still wont make its own unique connection to said site as it will already have the robots.txt on file, and thus leaving the connection purely down to the user agent of the user. There just wont be a user agent of Phorms to block.
---------- Post added at 17:58 ---------- Previous post was at 17:56 ----------
Rather than use the word 'block' I should have used the word 'deny' really as robots.txt is a system of honour and respecte rather than access protection.
|
I can specify "User-agent: Googlebot" and a specific rule in my robots.txt, it's BT are telling us there's implied consent and that site operators can opt out. We need a UA to do that effectively. Not every site has access to robots.txt if they want to continue down this "implied consent" route, they need to append the UA string for requests accordingly.