What is a Webcrawler?

[lucy2u]

I know this is probably a dumb question, but what is a Webcrawler? I've heard the term here and there and I've picked up on the fact that it has to do with search engines, but what exactly is it?

[larrysstyle]

A Webcrawler or as it's also known a Web Spider is a program that automatically fetches Web pages. Spiders are used to feed pages to search engines. It's called a spider because it crawls over the Web.

[carl owen]

Originally Posted by lucy2u

I know this is probably a dumb question, but what is a Webcrawler? I've heard the term here and there and I've picked up on the fact that it has to do with search engines, but what exactly is it?

A Web Crawler is basically an Automated program which is used to crawl on the Internet. Other terms for Web Crawler is bots, worms, spiders, robots etc. As these are automated programs, they can work in co-ordination with each other and Search Engines make the use of them to rank a webpage or website in the Search engine for Searches. They are programmed in such a way that they can also fetch details about the programming languages which are used to make the website or webpage.

As they are automated, they are well aware about the time they should visit a webpage or website. They also visit the Hyperlinks in your website. Hence, Hyperlinks are used in webpages and websites.

These Web crawlers are not only used by Search Engines but by some organizations and spammers also to fetch emails from the web pages and they send emails to these addresses which are not legitimate emails i.e. Spams.

Hope this helps