Crawl-Anywhere

Crawl-Anywhere

From the webpage:

April 2013 – Starting version 4.0, Crawl-Anywhere becomes an open-source project. Current version is 4.0.0-alpha

Stable version 3.x is still available at http://www.crawl-anywhere.com/

(…)

Crawl Anywhere is mainly a web crawler. However, Crawl-Anywhere includes all components in order to build a vertical search engine.

Crawl Anywhere includes :

Project home page : http://www.crawl-anywhere.com/

A web crawler is a program that discovers and read all HTML pages or documents (HTML, PDF, Office, …) on a web site in order for example to index these data and build a search engine (like google). Wikipedia provides a great description of what is a Web crawler : http://en.wikipedia.org/wiki/Web_crawler.

If you are gathering “very valuable intel” as in Snow Crash, a search engine will help.

Not do the heavy lifting but help.

Comments are closed.