Using the Meta robots tag
One of the ways to restrict search engine spiders from indexing pages that you don’t want indexed like duplicate content, dynamic pages, or development pages is by using the Meta robots tag. This tag can be placed in the head of your document and will identify a couple of things, whether the spider should index the page, and whether it should follow the page links.
Using the Meta robots tag is simple HTML. The tag looks like: <meta name="robots" content="index, follow" />. What this means is the tag is specified for the robots that visit that page, and this content should be indexed and its links followed. If you wanted to exclude this page from the search engine spiders then your content tag would be “noindex, nofollow” or any other combination of these commands.
It is important to note that using the Meta robots tag does not even remotely ensure that the search engines will do what you ask. In fact, more likely than not the search engines would follow any exclusion requests but not any to index or follow.
Featured Links