Thread: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
View Single Post
Old 08-05-2008, 14:46   #6072
SMHarman
Inactive
 
Join Date: Jun 2003
Services: Cablevision
Posts: 8,305
SMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronze
SMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronzeSMHarman is cast in bronze
Re: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
Quote:
Originally Posted by jelv View Post
Do we actually know if it is Google they look for in robots.txt or is it the any agent string? I posed this question a bit back and (unless I missed it) there was no answer:


If (unlikely as it is) they do obey the robots.txt rules we need a robots.txt file putting together which includes all the known valid agents and barring *
They are spoofing the site you are visiting to pass cookies. Do you think that they would not spoof the googlebot robot rather than name their robot their own.

In any case there is no robot involved in the activity of Phorm it is your browser so the agent is not googlebot but MSIE7.x or whatever you use and the result passed back to the L7 switch and your machine.

EDIT
OK so just read post 6060, so they will take the reply stream from the 3rd party site and will not index it until they have also got the robots.txt from that site under a separate request.
SMHarman is offline