Detecting Emergent Conflicts with Recorded Future + Ushahidi by Ninja Shoes. (?)
From the post:
An ocean of data is available on the web. From this ocean of data, information can in theory be extracted and used by analysts for detecting emergent trends (trend spotting). However, to do this manually is a daunting and nearly impossible task. We in this study we describe a semi-automatic system in which data is automatically collected from selected sources, and to which linguistic analysis is applied to extract e.g., entities and events. After combining the extracted information with human intelligence reports, the results are visualized to the user of the system who can interact with it in order to obtain a better awareness of historic as well as emergent trends. A prototype of the proposed system has been implemented and some initial results are presented in the paper.
The paper in question.
A fairly remarkable bit of work that illustrates the current capabilities for mining the web and also its limitations.
The processing of news feeds for protest reports is interesting, but mistakes the result of years of activity as an “emergent” conflict.
If you were going to capture the data that would enable a human analyst to “predict” the Arab Spring, you would have to begin in union organizing activities. Not the sort of thing that is going to make news reports on the WWW.
For that you would need traditional human intelligence. From people who don’t spend their days debating traffic or reports with other non-native staffers. Or meeting with managers from Washington or Stockholm.
Or let me put it this way:
Mining the web doesn’t equal useful results. Just as mining for gold doesn’t mean you will find any.