In Data Mining and Open APIs, Toby Segaran offers several reasons why data mining is important, including:
Human Cognition is Limited (slide 7)
We have all seen similar claims in data mining/processing presentations and for the most part they are just background noise until we get to the substance of the presentation.
The substance of this presentation is some useful Python code for several open interfaces and I commend it to your review. But I want to take issue with the notion that “human cognition is limited,” that we blow by so easily.
I suspect the real problem is:
Human Cognition is Unlimited
Any data mining task you can articulate could be performed by underpaid and overworked graduate assistants. The problem is that their minds wander from the task at hand, to preparation for lectures the next day, to reading assignments to that nice piece of work they saw on the way to the lab and other concerns. None of which are distractions that trouble data mining programs or the machines on which they run.
What is really needed is an assistant that acts like a checker with a counter that simply “clicks” as the next body in line passes. Just enough cognition to perform the task at hand.
Since it is difficult to find humans with such limited cognition, we turn to computers to take up the gauge.
For example, campaign contributions in the United States is too large a data set for manual processing. While automated processors can dutifully provide totals, etc., they won’t notice, on their own initiative, checks going to Illinois senators and presidential candidates from “Al Capone.” The cognition of data mining programs is bestowed by their creators. That would be us.