From BC$ MobileTV Wiki
Revision as of 07:37, 2 August 2019 by Bcmoney (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Robots is a text file webmasters can place within any directory or subdirectory/subdomain in order to specify whether or not the contents of the directory should be indexable by Search Engines, Recommendation Engines, Web Census', automated data extraction/submission scripts, Semantic Agents and other Web Bots.

Robots specifications should be placed in a file called:


within the directory which you wish to exclude from bots (a robot in this context is an automated script, web server or software which can access web pages automatically/programmatically)




External Links

See Also

Humans | Search Engine | Search | Crawler | Scraper | Robot