robots.txt to Exclude Content from Searches

Attracting spiders and bots to your website is one of your strategies to increase page rank and bring in more potential customers. But do you have pages of content you’d rather hide from those nosey search engines? Maybe you have multiple versions of a page on your site so you can split-test for effectiveness, or maybe you have different pages for viewing in the browser and one that is more printer-friendly. Rather than having them viewed as duplicate text, you can exclude one from being ‘crawled’.

Another aspect of many websites today is the value of privacy and sensitive information or data. It may be important to hide some of this information. You already know that search engines can only read text, so images and other types of graphics and javascript don’t really add any value to increasing your page rank. You may want to hide those from the creepy crawlers too. Read more