Thread: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
View Single Post
Old 13-05-2008, 07:20   #6424
jelv
Inactive
 
Join Date: Apr 2008
Posts: 128
jelv is an unknown quantity at this point
Re: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
Quote:
Originally Posted by AlexanderHanff View Post
Re: Robots.txt
Phorm claimed at the PIA Public Meeting that before they push a request through for a GET from a user to a website they will visit the document root for the domain to see if there is a robots.txt which allows Google access; if there is they will profile the pages the user requests. There is no indication (in fact they refused to tell us) what the user-agent will be for this robots.txt request ... <snip>
Alex: I have had confirmation from Emma Sanderson at BT that they will be checking robots.txt for googlebot - see quote from email in post here: http://www.cableforum.co.uk/board/34...-post6398.html
jelv is offline