[davews]

As Robert has already pointed out, using robots.txt as permission to profile is hopeless. Many sites do not use robots.txt and others, like we with ISP hosted sites, are unable to set one up anyway. If you are quite happy for Google to scan our sites and you therefore have no specific reason to include a robots.txt, then you won't bother. I imagine there are countless sites out there, including some significant ones, who have never bothered with it, or in some cases don't even know what it is.

It would be nice to suggest that if robots.txt were missing then Phorm would not profile that site, but I guess that would be asking too much.....

On a similar theme, Phorm have never made clear if they look at and use the <meta> tags in sites. I would feel tempted to include loads of words in those to totally upset the meaning of any profiling information, after all you only need to ensure they are the ten most used words on the site....