BI’s Dirty Secrets – The Unfortunate Domination of Manually-Coded Extracts by Rick Sherman.
From the post:
Manually-coded extracts are another dirty secret of the BI world. I’ve been seeing them for years, in both large and small companies. They grow haphazardly and are never documented, which practically guarantees that they will become an IT nightmare.
How have manually-coded extracts become so prevalent? It’s not as if there aren’t enough data integration tools around, including ETL tools. Even large enterprises that use the correct tools to load their enterprise data warehouses will often resort to manually-coded extracts to load their downstream BI data sources such as data marts, OLAP cubes, reporting databases and spreadsheets.
I thought the following passage was particularly good:
….Tools are easy; concepts are harder. Anyone can start coding; it’s a lot harder to actually architect and design. Tool vendors don’t help this situation when they promote tools that “solve world hunger” and limit training to the tool, not any concepts.
I don’t see manual coding as a problem, so long as it is documented. There should be one and only one penalty for lack of documentation. Termination.
Lack of documentation can put critical IT systems at risk and it doesn’t take complex systems to produce. Even (gasp) MS Word documents that are maintained with a table of contents and indexes can be adequate documentation.
Not the same as a bug database with bug reports, patches, pointers to code, email discussions, meeting minutes, etc., but interactive production of graphs and charts isn’t a requirement for successful documentation.
Undocumented manually-coded extracts are a sign that the requirements of BI users are not being meet. Getting those documented and incorporated into BI tools looks like a good first start to solving this secret.