Quote:
Originally Posted by R Jones
Interesting discovery relating to my BTOpenworld free ISP space.
snip
Then I discovered the problem - I can't actually set up a "valid" robots.txt for the ISP hosted pages - because as far as google is concerned the only robots.txt it sees is the one at the top level domain - the ISP domain of btinternet.com .
My site robots.txt is www.btinternet.com slash tilde~ username slash robots.txt
The one google sees is www.btinternet.com/robots.txt which merely says
User-agent: *
Disallow: /Templates
Disallow: /virtualworlds
which seems to be the one that btinternet.com uses for all their hosted space. |
Exactly the same applies to all the Yahoo geocities sites as well - just checked. Google simply returns a 404 when asked to look for robots.txt on a google verified BTYahoo! geocities site that I just created, and for which I uploaded a robots.txt file.
Geocities is the current ISP free webspace available to BTYahoo! customers. BTOpenworld is the legacy space which older customers like me still have but is not available to new customers.
I imagine the same applies to the wider Yahoo/Geocities sites too which must number millions and are by and large, personal pages run by families.
Both sorts DON'T allow a website owner to modify robots.txt. So that is the entire BT customer base of free webspace that doesn't actually work with one of the legal workarounds they are trying to convince us about, in the Webwise/Phorm model they are promoting.