About Robots.txt

The robots.txt file tells search engines what they should and should not index (make available as links).

You can simply create a blank file and name it robots.txt. (This will reduce site errors and allow all search engines to rank you if they want.)

 

To create a robots.txt file, use any text editor (notepad) to create a new document. To allow all search engine bots into your site, use the following text:

 

User-agent: *
Disallow:

 

Save the file as robots.txt and then upload it into the www root of your website (public_html).

If you want to stop search engines from ranking you, use this code:

#Code to not allow any search engines!
User-agent: *
Disallow: /

 

You may use robots.txt to allow/disallow specific folders and files. See more codes here: http://www.robotstxt.org/robotstxt.html.

 

Another great resource is http://tool.motoricerca.info/robots-checker.phtml which is a robots.txt validator and will tell you if your robots.txt file is written properly.

 

One extra thing to note is that you should not use robots.txt to try to hide information you want kept private. robots.txt is viewable by anyone so you will telling people where to look, even though the information should not show up in search engines. Additionally, there are bots/crawlers that will ignore your robots.txt file, these are mostly used by spammers.