Originally Posted by pseudonym

As they will be caching robots.txt, wouldn't it be better to deliver phorm a massive list of disallow statments, including for paths that do not exist - storage may be cheap, but searching it would make a bit more work for Phorm's profilers every time a phorm victim requested a page from your site?