Quote:
Originally Posted by tarka
This has just given me an idea, although not exactly a straightfoward proposition.
If/When the BT trial happens, it's not going to be that difficult to work out what that user agent will be, just visit your own website (assuming you are phormed) then check your web logs.
Now assuming they don't forge a googlebot user agent and do use their own unique user agent, then it should be fairly simple to configure a web server to parse robots.txt as a script (I am sure I could set this up easily with apache/php) and serve different content based on the user agent. If it's a phorm user agent then deny the entire site, if not then serve your usual robots.txt.
Although this still doesn't get round the implied consent/default opt-in issue for webmasters/content authors, it's something to think about.
Regards...
T
|
I don't for one minute think Phorm would honour robots.txt if it explicitly denies them access. This is exactly why they won't tell us what user-agent they plan to use because they don't want to be denied access.
Let's not forget that robots.txt is not an access control mechanism, it is an honour based system which robots can either adhere to or ignore, it doesn't physically stop them accessing pages.
If their user-agent ever does get discovered, it would be useful to just add a script to your site which checks user-agent and if the Phorm user-agent is detected it builds a page which says something like "Get your hands of me you dirty ape!" or "Phorm is not welcome here, please go away." etc etc etc.
Alexander Hanff