This is just one example of any number of articles that lament how hard Hadoop is to explain to non-technical users.
Apparently there is an anticipated flood of applications that will have Hadoop “under the hood” so to speak that are due out later this year.
While I don’t doubt it will be true that enormous amounts of data will be analyzed by those applications, without some underlying understanding of the data, will the results be meaningful?
Note that I said the data and not Hadoop.
Understanding Hadoop is just a technicality.
An important one but whether one uses a cigar box with paper and pencil or the latest non-down cloud infrastructure with Hadoop, understanding the data and the operations to be performed upon it are far more important.
Processing large amounts of data will not be cheap and so the results of necessity will be seen as reliable. Yes? Or else we would not have spent all that money and you can see the answer to the problem is….
You can hear the future conversations as clearly as I can.
BigData simply means you have a big pile of data. (I forego the use of the other term.)
Whether you can extract meaningful results depends on the same factors as before the advent of “BigData.”
The principal one being an understanding of the data and its limitations. Which means human analysis of the data set and its gathering.
Data sets (large or not) are typically generated or used by staff and capturing their insights into particular aspects of a data set can be easily done using a topic map.
A topic map can collect and coordinate multiple views on the use and limitations of data sets.
Subsequent users don’t discover too late that a particular data set is unreliable or limited in some unforeseen way.
Hadoop is an important emerging technology subject to the rule:
BigGarbage In -> BigGarbage Out.