The Scourge of Data Silos by Rick Sherman
From the post:
“Those who cannot remember the past are condemned to repeat it.” [1]
Over the years there have been many technology waves related to the design, development and deployment of Business Intelligence (BI). As BI technologies evolved, they have been able to significantly expand their functionality by leveraging the incredible capacity growth of CPUs, storage, disk I/O, memory and network bandwidth. New technologies have emerged as enterprises’ data needs keep expanding in variety, volume and velocity.
Technology waves are occurring more frequently than ever. Current technology waves include Big Data, data virtualization, columnar databases, BI appliances, in-memory analytics, predictive analytics, and self-service BI.
Common Promises
Each wave brings with it the promise of faster, easier to use and cheaper BI solutions. Each wave promises to be the breakthrough that makes the “old ways” archaic, and introduces a new dawn of pervasive BI responsive to business needs. No more spreadsheets or reports needed!
IT and product vendors are ever hopeful that the latest technology wave will be the magic elixir for BI, however, people seem to miss that it is not technology that is the gating factor to pervasive BI. What has held back BI has been the reluctance to address the core issues of establishing enterprise data management, information architecture and data governance. Those core issues are hard and the perpetual hope is that one of these technology waves will be the Holy Grail of BI and allow enterprises to skip the hard work of transforming and managing information. We have discussed these issues many times (and will again), but what I want to discuss is the inevitable result in the blind faith in the latest technology wave.
Rick does a good job at pointing out “the inevitable result in the blind faith in the latest technology wave.”
His cool image of silos at the top is a hint about his conclusion:
I have railed about data silos, along with everyone else, for years. But the line of data silos seems to be endless. As indeed I have come to believe it is.
Endless that is. We can’t build data structures or collections of data without building data silos. Some times with enough advantages to justify a new silo, sometimes not.
Rather than “kick against the bricks” of data silos, our time would be better spent making our data silos as transparent as need be.
Not completely and in some cases not at all. Simply not wrote the effort. In those cases, we can always fall back on ETL, or simply ignore the silo altogether.
I posted recently about open data passing the one millionth data set. Data that is trapped in data silos of one sort or another.
We can complain about the data that is trapped inside or we can create mechanisms to free it and data that will inevitably be contained in future data silos.
Even topic map syntaxes and/or models are data silos. But that’s the point isn’t it? We are silo builders and that’s ok.
What we need to add to our skill set is making windows in silos and sharing those windows with others.