How Computers Broke Science — and What We can do to Fix It by Ben Marwick.
From the post:
Reproducibility is one of the cornerstones of science. Made popular by British scientist Robert Boyle in the 1660s, the idea is that a discovery should be reproducible before being accepted as scientific knowledge.
In essence, you should be able to produce the same results I did if you follow the method I describe when announcing my discovery in a scholarly publication. For example, if researchers can reproduce the effectiveness of a new drug at treating a disease, that’s a good sign it could work for all sufferers of the disease. If not, we’re left wondering what accident or mistake produced the original favorable result, and would doubt the drug’s usefulness.
For most of the history of science, researchers have reported their methods in a way that enabled independent reproduction of their results. But, since the introduction of the personal computer — and the point-and-click software programs that have evolved to make it more user-friendly — reproducibility of much research has become questionable, if not impossible. Too much of the research process is now shrouded by the opaque use of computers that many researchers have come to depend on. This makes it almost impossible for an outsider to recreate their results.
Recently, several groups have proposed similar solutions to this problem. Together they would break scientific data out of the black box of unrecorded computer manipulations so independent readers can again critically assess and reproduce results. Researchers, the public, and science itself would benefit.
Whether you are looking for specific proposals to make computed results capable of replication or quotes to support that idea, this is a good first stop.
FYI for business analysts, how are you going to replicate results of computer runs to establish your “due diligence” before critical business decisions?
What looked like a science or academic issue has liability implications!
Changing a few variables in a spreadsheet or more complex machine learning algorithms can make you look criminally negligent if not criminal.
The computer illiteracy/incompetence of prosecutors and litigants is only going to last so long. Prepare defensive audit trails to enable the replication of your actual* computer-based business analysis.
*I offer advice on techniques for such audit trails. The audit trails you choose to build are up to you.