Robots.txt

Zohann · #1 (**permalink**) 12-12-09, 08:02

I have found this file in the root of my web site that has the following code in it:

User-agent: *
Disallow:

What does this mean?

CustomServer · #2 (**permalink**) 12-12-09, 08:29

Robots.txt is used to control the way search engine robots visit your site for indexing purposes.

Code that you found states that all parts of the site are allowed for robots to see and index.

Jonathan · #3 (**permalink**) 12-12-09, 08:49

Actually you don't even need that code because by default search engine robots, or in short robots, will index your site. But if you put:
User-agent: *
Disallow: /
it will block robots from your entire site.

Crazzy · #4 (**permalink**) 12-12-09, 09:04

Having site indexed is the most important thing you need to ensure. Having that code in the root of your site is OK, but if you don't know what code is inside this file it's better to erase the file than to have some code that may prevent robots from indexing it.

Zohann · #5 (**permalink**) 12-12-09, 09:29

I don't like the clutter, so I took you up on your advice and removed the file completely. And of course I realize that indexing is very important.

HelpHolly · #6 (**permalink**) 12-18-09, 14:36

Are there any true reasons why someone would not want robots indexing their site? The robots are what help you get the page rank and search stats right?

Caddie · #7 (**permalink**) 12-19-09, 10:21

I'm curious to know the answer to Holly's question. I have always thought that allowing robots access to every bit of information was critical to getting a good PR rank.

Kratos · #8 (**permalink**) 12-26-09, 13:43

This is done to stop search engines accessing private or irrelevant sections of your website or at the early development of a website if the client wants to keep the site secret until official launch date.

NCHosting · #9 (**permalink**) 12-29-09, 06:30

Thats right.

Google will crawl for everything it can.
Your running a site you don't want advertised, perhaps its an admin server or such.
Even something a little on the left of the law :p

It's there so YOU have control over what happens with your website.

Fletcher · #10 (**permalink**) 12-29-09, 10:20

Robot.txt contains some instructions for the robots which crawls your link. This is very important for a good site

bluetrance05 · #11 (**permalink**) 03-09-10, 01:29

The code means they gives access to all robots to all of your site. But if you disallow it, I think a "/" sign must be placed after the Disallow: label.

Zohann · #12 (**permalink**) 09-21-10, 02:04

Quote:

Originally Posted by Kratos

This is done to stop search engines accessing private or irrelevant sections of your website or at the early development of a website if the client wants to keep the site secret until official launch date.

I agree. There are some things that you just don't want to share with the general public.