From the webpage:
April 2013 – Starting version 4.0, Crawl-Anywhere becomes an open-source project. Current version is 4.0.0-alpha
Stable version 3.x is still available at http://www.crawl-anywhere.com/
(…)
Crawl Anywhere is mainly a web crawler. However, Crawl-Anywhere includes all components in order to build a vertical search engine.
Crawl Anywhere includes :
- a Web Crawler with a Web administration interface (http://www.crawl-anywhere.com/crawl-anywhere/)
- a document processing pipeline (http://www.crawl-anywhere.com/simple-pipeline/)
- a Solr indexer
- a Solr tags cloud analyzer
- a full featured and customizable Web search application (some search engines using Crawl-anywhere : http://www.hurisearch.org/ or http://www.searchamnesty.org/)
Project home page : http://www.crawl-anywhere.com/
A web crawler is a program that discovers and read all HTML pages or documents (HTML, PDF, Office, …) on a web site in order for example to index these data and build a search engine (like google). Wikipedia provides a great description of what is a Web crawler : http://en.wikipedia.org/wiki/Web_crawler.
If you are gathering “very valuable intel” as in Snow Crash, a search engine will help.
Not do the heavy lifting but help.