A robots.txt file is plain-text file that allows Web site owners to control the way search engine spiders crawl and/or index their Web sites/pages. The standard method of using this format is to create and insert the robots.txt file in the root directory of the Web site in question. A robots.txt file enables you to command all or certain spiders to steer clear of any part or all pages and files on a given Web site. Once uploaded, visiting spiders will open and read the robots.txt file and implement its directions before crawling the affiliated Web site.
The robots.txt file consists of two defining elements
user-agent
disallow /
The first element User-agent specifies which agents, spiders or browsers should read and obey the commands in the file. An asterisk () denotes all spiders. The second element Disallow defines which files and directories should be blocked from the search engines.
The robots.txt file must be placed in the server's root directory; it cannot simply be put in a user's personal subdirectory space.
Example of proper placement of robots.txt file www.domainnamgoeshere.com/robots.txt
Example of useless placement of robots.txt file www.domainamegoeshere/admin/robots.txt
If you cannot access the root directory, you can instead use the robots Meta tag format to direct the spiders' behavior. Note that this tag does not allow users to target specific spiders. Whereas a robots.txt file can be used to control spider behavior throughout an entire Web site, a separate robots Meta tag must be inserted into each applicable Web page.
