Quote:
Originally Posted by oblonsky
I trust that the ISPs will ensure that Phorm complies with robots.txt. Their argument for implied consent is weak already but disappears entirely if webmasters are serving them a DENY ALL robots.txt.
|
I took a look at
iplists.com/ and did some reverse lookups, it doesn't appear that PTR's are consistent for all the netblocks listed there. There'd be issues even with a cidr whitelist, some of the listed IP's are in blocks assigned to savis, bejing telecom and BT Fusion.
Incidentally, BT wouldn't be planning to buy cached robots.txt data from a certain search partner would they?
Quick CIDR whitelist example