Quote:
Originally Posted by mattyh
Coming into this a bit late I realise... but several pages back, some were saying that Google relied on implied consent to crawl websites...
Don't you have to physically add each site you wish them to index to their list of URLs, at www.google.com/addurl
therefore inphormed consent is gained, by the act of you adding your site to their index?
Or have I got confused?  |
Google runs 'webcrawlers' which follow links to find and index websites. So if you have a site that is linked from anywhere eventually google is going to find it and index it. If it finds a robots.txt file the webcrawler will obey its instructions about what and what not to index. The URL you quote can be used to encourage the webcrawler to visit your site sooner rather than later, most website owners welcome this.
