Robots.txt & User-agent:

What is robot text, how does it look like and how can you use it?
Webmasters and Search Engine optimization companies make standard use of this meta tag. Some people call them "Fresh Tags". They are very useful and can help you to get more often crawled by the bots and spiders.

You can place them were you want in your meta data section, which is located between the header section of your HTML code.

Meta tags can exclude all outgoing links on a page, but you can also command Googlebot not to follow individual text links by adding rel="nofollow" to a hyperlink.

When Googlebot sees the attribute rel="nofollow" on text links, those links won't get any votes when Google ranks the web sites for the search results.

Example, if you would exchange links with SEO WATCH:

<a href=http://www.SEO-watch.com/>SEO for Google Marketing </a>

You could replace it with:

<a href=http://www.SEO-watch.com/ rel="nofollow">SEO for Google Marketing </a>.

The content of the robots Meta Tag contains directives separated by commas.

The currently defined directives are [NO]INDEX and [NO]FOLLOW.
The INDEX directive specifies if an indexing robot should index the page. The follow directive specifies if a robot is to follow links on the web page.

The defaults are INDEX and FOLLOW.
The values all and none set all directives on or off: ALL=INDEX,FOLLOW and NONE=NOINDEX,NOFOLLOW.

You can place as an alternative the robots text into the root of your server, to control which of your web pages will be listed in the databases of the Search Engine results and which directories are forbidden for the crawler or bot. Some stuff you do not want publicly displayed and when not restricting the robots it will get public. You should save the text file as robots.txt and insert the following lines to forbid specific files and directories, the below record describes the default access policy for any robot symbolized by the *:

Disallow: /cgi-bin
Disallow
The value of this field specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example: