Robots
From BC$ MobileTV Wiki
Robots is a text file webmasters can place within any directory or subdirectory/subdomain in order to specify whether or not the contents of the directory should be indexable by Search Engines, Recommendation Engines, Web Census', automated data extraction/submission scripts, Semantic Agents and other Web Bots.
Robots specifications should be placed in a file called:
robots.txt
within the directory which you wish to exclude from bots (a robot in this context is an automated script, web server or software which can access web pages automatically/programmatically)
Tools
- Robots.txt - Analyzer: http://tools.seobook.com/robots-txt/analyzer/ (verify accessibility of URLs compared to robots.txt rules)
- Robots.txt syntax checker: http://www.sxw.org.uk/computing/robots/check.html
Resources
- The original specification for Web Robots files: http://www.robotstxt.org/orig.html | EXAMPLES
- GOOGLE - changing your site's title and description in search results: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=35264
Tutorials
- How to Create & Analyze Robots.txt Files: http://tools.seobook.com/robots-txt/
- robots.txt allow all except few sub-directories: https://stackoverflow.com/questions/28495972/robots-txt-allow-all-except-few-sub-directories#28526427
- What is a robots.txt file?: https://moz.com/learn/seo/robotstxt
External Links
- wikipedia:Robots.txt
- The Robots META Tag Values: http://www.searchtools.com/robots/robots-meta.html
- What are the Meta Tags ‘NOODP’ and ‘NOYDIR’ Used For in SEO?: http://www.seoboy.com/what-are-the-meta-tags-noodp-and-noydir-used-for-in-seo/
- Tests of Robots Following Rules: http://www.searchtools.com/test/robots/index.html
- Google supports META NOODP tag: http://www.mattcutts.com/blog/google-supports-meta-noodp-tag/
- Get rid of those directory snippets using NOODP & NOYDIR (links to instructions for Bing,Yahoo,Google): http://www.seoworkers.com/seo-articles-tutorials/using-noodp-and-noydir.html
- How to block ALL bots?: https://www.webmasterworld.com/robots_txt/3769121.htm
