was on digg and looked at this article on self made webspiders
http://www.ibm.com/developerworks/li...der/index.html
I can only hope the below will come around and bite spamphorm on the ass.. bt included.
Jtech
(taken from the site)
Legal issues
There have been lawsuits for data mining on the Internet using Web spiders, and they've not gone well. Farechase, Inc. was recently sued by American Airlines for screen scraping (done in real-time). The lawsuit first claimed that gathering the data violated American Airlines' users' agreement (found under Terms and Conditions). When that wasn't successful, American Airlines claimed a form of trespass, which was successful. Other lawsuits claim that the bandwidth taken by the spiders and scrapers detracts from legitimate users. All are valid claims and make politeness policies all the more important. See the Resources section for more information.