Rebots.txt file and Importance Of Search Engine

[Shane Melson]

In order to Create Robots.txt file, please make a note of the following -

Robots.txt file can be created in Notepad -

User-Agent: [Spider or Bot name]
Disallow: [Directory or File Name]

If you need to excluse any file from the search engine, you may simply enter the following -

User-Agent: Googlebot
Disallow: /private/privatefile.htm

In order to disallow a section from SE -

User-Agent: *
Disallow: /newsection/

In order to allow SE to index everything on site, enter the following -

User-agent: *
Disallow:

To disallow any SE to index your site, enter the following -

User-agent: *
Disallow: /

[carl]

Robots.txt file plays a very important role in SEO.
This robots.txt file is a powerful tool when it used correctly, but if it is used incorrectly it can leave your private data exposed to the public and also effects your search engine rankings.

Note:You sholud use these robots.txt file only when necessary and make sure That you use it correctly.

[aliya]

Shane can you Explain us in details how it is useful

[Shane Melson]

It is not possible for a human to index every website on the website and rank them accordingly. In order to cover the cost of time and man power. This is a concept brought in by the Search engines.

Every website indicates the bots of those search engine on how to index the website and which section they should not index it.

This is hence a important concept in relation of SEO techniques.

[aliya]

Thanks Shane for explaining the whole concept

[Shane Melson]

Your Welcome Aliya

[MR. Maniac]

Great post Shane

[Shane Melson]

Have you tried it. That is pretty old concept.

[aliya]

It may be old concept for you but some din't even knew what is it

[linux tech]

Perhaps its not linux atleast. Now you can remember that and shouldnt be a problem for you.