|
Re: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
With regards to user agents and robots.txt it is important to note that there are 3 user agent strings involved!
1. The user agent string sent in headers for normal page requests (likely to be unaltered from the users)
2. The user agent string sent in the headers for the request of robots.txt (no idea what this will be)
3. The user agent (if any) used to match lines in the robots.txt file. (Maybe googlebot but who knows??)
|