[rryles]

I'm under the impression that RIPA requires consent from both parties in a communication. I'm trying to see how that could be satisfied for site owners using robots.txt. I don't think it's possible. In fact I don't think it's possible with any other mechanism either. The http request would have to be intercepted to see which domain it was for before it could be checked if that domain has consented to interception. The destination IP address isn't even enough as multiple domains can be served from a single IP. You have to look at the actual http headers to find the domain.

For those who are less up on the inner workings of the internet I'll try and make an analogy.

Suppose you have consented to have your calls intercepted by BT(Just imagine you did). You make a phone call to a telephone number that is used by several people. Some of these people have also consented to interception, but not all. In order for BT to know if the other party has consented to interception they have to intercept the call and listen in until you ask to speak to bob (or whoever it may be). They can't do that until they know who is on the other end though so it's catch 22.