I now know the user agent that Phorm is looking for in robots.txt and its time to get Google's opinion methinks. I have received confirmation from BT that Phorm is explicitly looking for the Googlebot's permission in robots.txt - not any search engine:
I posed the following question in an email to Emma Sanderson:
Quote:
2. Could you provide clear guidance on how robots.txt is used. We keep having Google cited as an example. Does this mean that Phorm pretends to be Google and obeys the Google rules or does to obey the normal rules and only profile visitors to a site if User-agent: * is allowed. For example if a robots.txt was in place which specifically allowed Google only and barred all other agents, i.e.
User-agent: Google
Disallow:
User-agent: *
Disallow: /
would visitors be profiled by Phorm?
If it does not obey the normal rules and pretends to be Google I'm sure that company will have something to say about it!
|
(Note I whoopsied and got the name of the Google agent wrong)
This evening I have received a reply which included the following:
Quote:
With regard to the second point you raise in your email dated 8 May 08, I have cut and paste your example/question and added comments (in blue)
User-agent: Googlebot
Disallow:
User-agent: *
Disallow: /
would visitors be profiled by Phorm?
Yes because Googlebot is allowed.
If it does not obey the normal rules and pretends to be Google I'm sure that company will have something to say about it!
The Phorm check does not 'pretend' to be Googlebot, it uses the same permissions are were set for Googlebot.
Regards
Emma
|
The fact that they have corrected it to Googlebot confirms that it is specifically looking for the Googlebot permission.