Certain types of Web pages typically have little to no value in a search index. Examples include: "Document Not Found" (404) error pages, comment or reply forms for blog entries, and "printer versions" of news articles. A comment form is generic and meant for data input rather than output to the reader, and the printer version of an article only duplicates content already indexed in the regular version.
You can use META tags in an HTML document to prevent the Search Appliance adding the document to its search index or to prevent the crawler following links from the document to other documents. The META "name" attribute is "robots", and the "content" attribute must contain "noindex", "nofollow", or both:
<meta name="robots" content="noindex" />
<meta name="robots" content="nofollow" />
<meta name="robots" content="noindex,nofollow" />
Note: some browsers do not display the code snippet above, so here's an image of the text.
