Robots

From BC$ MobileTV Wiki
Jump to: navigation, search

Robots is a text file webmasters can place within any directory or subdirectory/subdomain in order to specify whether or not the contents of the directory should be indexable by Search Engines, Recommendation Engines, Web Census', automated data extraction/submission scripts, Semantic Agents and other Web Bots.

Robots specifications should be placed in a file called:

 robots.txt

within the directory which you wish to exclude from bots (a robot in this context is an automated script, web server or software which can access web pages automatically/programmatically)




Tools


Resources


Tutorials

External Links

See Also

Humans | Search Engine | Search | Crawler | Scraper | Robot