your business online

Robots Text File

Like the Sitemap.xml and .htaccess file you can create a robots text file - robots.txt simply using Notepad.

Create your file something like this shown below and save it as a text file.

#
# robots.txt for http://www.your-business-online.com
#
# exclude some access-controlled areas
User-agent: *
Sitemap: http://www.your-business-online.com/sitemap.xml
Disallow: /formmail
Disallow: /hidden
Disallow: /javascript
Disallow: /logs
Disallow: /php
Disallow: /ServerTest
Disallow: /style
User-agent: ia_archiver
Disallow: /

In the above robots.txt file we tell Search engine spiders what folders they can't look into. We tell them where our sitemap is located so they should be able to find all the pages of our site that we want them to find. At the bottom we've disallowed one robot which stops it archiving our site. Using the robots.txt file you can "ask" certain spiders not to crawl your site if you so wish. Many people think that the robots text file will stop all unwanted robots from visiting your site - but that simply isn't true. The ia archiver robot is a "good robot" but we didn't particular want our sites entire history listed on another site.