Title & Meta Tags     

Robot

nofollow

noindex

<Meta Name="Robots" content="noindex">
Place this Meta Tag in the head of your document if you do not want the robot to follow any links on the page

Multiple Pages

If you have a lot of pages that you want to protect you can place a robots text file in the root directory.

• The robots.txt file must be placed in the root directory.
• The file name needs to be lower case: robots.txt
• You need to place the following information at the top of the page
  (give the name of a specific search engine robot or for all robots list an asterisk)
• User-Agent:*
• Disallow: (list the pages you want to block)
• You need to set the permissions on the file so that the search engines can read it.
• You can also disallow the search engine from indexing images and other elements
  

Allows everyone everywhere

User-agent: *
Disallow:




Keeps everyone out

User-agent: *
 Disallow: /


Disallows cgi-bin and images

User-agent: *

Disallow: /cgi-bin/ Disallow: /images/


Google with allow an asterisk in the Disallow but most search engines will not

User-agent: Googlebot-Image
Disallow: /*.gif$


Allows only major search engines
You would continue the list. Below is just the robots for Google and Yahoo


User-agent: Googlebot

 

Disallow:

User-agent: Slurp

* Disallow: /

NoODP Metatag

This is relatively new. This Metatag allows you to tell MSN and Google that you do not want them to consider using the Open Directory Project (ODP) description for your site as the description they show in their search results.