A spider is a small web bot type program that looks for a small file on your webspace/server called robot.txt.
It uses this file to index your web pages based on things like keywords,title,description and content.
Using these effeciently you can get your website listed higher in the list order.
For example go to
www.google.com and type in chatroom danger
or just click
http://www.google.com/search?hl=en&i...hatroom+danger
Out of 13,500 websites you can see one of my sub pages listed second.
This is becuase I have listed it correctly and made the most efficent used of keywords title, description and content.
The robots.txt file can be very useful in that you tell program the spider not to look or index certain pages from your site if you so wish to. For exampl you might want to protect certain directories from being cached in the google engine.
Very useful indeed.
More info
http://www.searchengineworld.com/rob...s_tutorial.htm
http://hotwired.lycos.com/webmonkey/96/51/index2a.html