Robots Text File
Create your file something like this shown below and save it as a text file.
In the above robots.txt file we tell Search engine spiders what folders they can't look into. We tell them where our sitemap is located so they should be able to find all the pages of our site that we want them to find. At the bottom we've disallowed one robot which stops it archiving our site. Using the robots.txt file you can "ask" certain spiders not to crawl your site if you so wish. Many people think that the robots text file will stop all unwanted robots from visiting your site - but that simply isn't true. The ia archiver robot is a "good robot" but we didn't particular want our sites entire history listed on another site.