What is Robots.txt?

999sam · #1 (**permalink**) 05-30-11, 03:57

Hi!
I want to know that What is Robots.txt? Please give your suggestion about it.

77wmaster · #2 (**permalink**) 05-30-11, 04:35

A robots.txt file is simply a text file that resides in your web site’s root directory, and it dictates what type of visitors can view your web site. This file is often utilized to block search engines or another robots from crawling the site and indexing the information. This file can also be utilized to block specific visitors based on their IP address. For example, if somebody is abusing your site, you may want to block their IP address from accessing the site utilizing the robots.txt file.

Some hosting accounts don't add a robots.txt file automatically, while another leave the file blank by default. The exact configuration of the file will depend upon the web hosting provider. It is always best to check if the robots.txt file is edited to disallow SE, as this could be very detrimental to your web site’s success.

999sam · #3 (**permalink**) 05-30-11, 05:12

What are the advantages of Robot.txt?

WebMaster · #4 (**permalink**) 05-30-11, 06:28

Just to let you know that a robots.txt file only tells well behaved spiders not to index (i.e exclude) parts of your website. The robots.txt file doesn't permit any form of security or protection. Many spiders out there completely ignore the robots.txt file and will index everything.

77wmaster · #5 (**permalink**) 05-30-11, 06:43

Yes, I agree with you.
There are some spammy robots that don't respect robots.txt instructions..if you have blocked them by robots.txt., they will still follow the whole pages of your site.

In this case you can track their IP addresses by your traffic log and then block them utilizing .htaccess... It will also assist you to cut down your bandwidth usage as half of your website bandwidth is utilized by these spammy bots if you not block them.

johnhardy009 · #6 (**permalink**) 06-08-11, 06:21

Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.In those cases, when you have a complex robots.txt file – i.e. you give different instructions to different user agents.