Whats this?

[ic14]

Guest
(Google Spider)


Hmm, an odd guest there, what does it mean?

[Mark W]

Originally Posted by ic14

Guest
(Google Spider)
Hmm, an odd guest there, what does it mean?

if you submit a website to google it will trawl through your pages to log it.

The ip it uses to visit your site is called a google spider AFAIK

look Here for a list of the IPs google uses to do this......

[MadGamer]

Cool how they do it though

There's a dozen of the blighters on the board right now! It's making me itchy, like I just walked through a cobweb ...

[Tiptoes]

A spider is a small web bot type program that looks for a small file on your webspace/server called robot.txt.

It uses this file to index your web pages based on things like keywords,title,description and content.

Using these effeciently you can get your website listed higher in the list order.

For example go to www.google.com and type in chatroom danger

or just click

http://www.google.com/search?hl=en&i...hatroom+danger


Out of 13,500 websites you can see one of my sub pages listed second.

This is becuase I have listed it correctly and made the most efficent used of keywords title, description and content.


The robots.txt file can be very useful in that you tell program the spider not to look or index certain pages from your site if you so wish to. For exampl you might want to protect certain directories from being cached in the google engine.

Very useful indeed.


More info

http://www.searchengineworld.com/rob...s_tutorial.htm

http://hotwired.lycos.com/webmonkey/96/51/index2a.html