Web spider

From BC$ MobileTV Wiki
Revision as of 02:57, 18 May 2021 by Bcmoney (Talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

A Web Spider (also commonly referred to as a Web Crawler) is a program which runs indefinitely and visits links on the web for the purpose of generating information about the underlying link-structure of the web. [1]



CommonCrawl is an attempt to create an open and accessible crawl of the web for education, research and other non-commercial innovation.


Sphider is a lightweight web spider and search engine written in PHP, using MySQL as its back end database. It is a great tool for adding search functionality to your web site or building your custom search engine. Sphider is small, easy to set up and modify, and is used in thousands of websites across the world.



[2] [3] [4]

External Links


  1. wikipedia:Web spider
  2. Multithreaded Webcrawler: https://sites.google.com/site/javagamescorner/standalone-tutorials/concurrency-1/multithreaded-webcrawler
  3. Implementing Threads Into Java Web Crawler: https://stackoverflow.com/questions/26363585/implementing-threads-into-java-web-crawler
  4. How to write a Multi-threaded WebCrawler: http://www.andreas-hess.info/programming/webcrawler/index.html

See Also

Scraper | Search Engine