Robots TXT File Format
The standard robots txt file format specified by the W3C which is the World Wide Web Consortium and supplies standard guidelines for web development is fairly straightforward. Basically this file should be named robots.txt and should be placed in the root directory of your website.
The robots txt file format is as follows.
The first line of the robots.txt file should indicate what it is:
# robots.txt for http://www.somesite.com
The next line indicates who the file pertains to including Google and MSN Bots or all.
User-agent: *
In this instance user agent is all indicated by the asterisk.
The next section is for sections or pages that should be excluded from indexing:
Disallow: /development
Disallow: /test.html
This is the basic robots txt file format that should be used on each website.
Featured Links