Insurance to protect a site against a potential penalty
A Robots.txt protocol is of great help to a webmaster or web site owner if done correctly. Sites on different topics providing vast information are available on the internet. Some sites are extremely well written but its not always that you get what you see.
Spamming the engines for better ranking has become a common gadget. Spam is getting ranking with keywords or key phrases used by the visitors. Spam uses different methods like hidden text, keyword stuffing in the meta tags, doorway pages and cloaking issues.
Search engines have now developed algorithms to trace the unethical methods used for search engine optimization. But to be away from the eyes of these algorithms unique folders or sub-directories and Robots.txt exclusion protocol can be used so that the search engine do not index the duplicate content.
Robots.txt exclusion protocol is rightly said to be the “insurance” against getting the site penalized or banned.