Analyzing 1.2 Million Network Packets per Second in Real Time by James Sirota and Sheetal Dolas.
Slides giving an overview of OpenSOC (Open Security Operations Center).
I mention this in case you are not the NSA and simply streaming the backbone of the Internet to storage for later analysis. Some business cases require real time results.
The project is also a good demonstration of building a high throughput system using only open source software.
Not to mention a useful collaboration between Cisco and Hortonworks.
BTW, take a look at slide 18. I would say they are adding information to the representative of a subject, wouldn’t you? While on the surface this looks easy, merging that data with other data, say held by local law enforcement, might not be so easy.
For example, depending on where you are intercepting traffic, you will be told I am about thirty (30) miles from my present physical location or some other answer. 😉 Now, if someone had annotated an earlier packet with that information and it was accessible to you, well, your targeting of my location could be a good deal more precise.
And there is the question of using data annotated by different sources who may have been attacked by the same person or group.
Even at 1.2 million packets per second there is still a role for subject identity and merging.