Thread: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
View Single Post
Old 13-05-2008, 01:42   #6419
AlexanderHanff
Permanently Banned
 
Join Date: Mar 2008
Posts: 1,028
AlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful oneAlexanderHanff is the helpful one
Re: Virgin Media Phorm Webwise Adverts [Updated: See Post No. 1, 77, 102 & 797]
Quote:
Originally Posted by Phormic Acid View Post
Did they specifically refer to the Googlebot by name? Most disallows use a user-agent of *, which would, of course, include Google. Google’s a reasonable example. A quote attributable to Phorn in Richard Clayton’s paper is far more general.
We work on the basis that if a site allows spidering of its contents by search engines, then its material is being openly published. Conversely, if the site has disallowed spidering and indexing by search engines, we respect those restrictions in robots.txt.
Why restrict themselves to only Googlebot, what about MSNBot and Slurp, etc.?
They only specified Google but I presume they were just being flippant. I expect if there is anything at all representing permission to spider to anyone, they will use that as implied consent.

Alexander Hanff
AlexanderHanff is offline