Quote:
Originally Posted by JackSon
Rather than use the word 'block' I should have used the word 'deny' really as robots.txt is a system of honour and respecte rather than access protection.
|
You need to adopt the new Parasites.txt industry standard inclusion protocol. Its the inverse of the Robots.txt exclusion protocol, and its deny directives. Parasites.txt uses allow directives.
See
parasitestxt.org.
I don't think Phorm have announced compliance yet, but they should.