Statistics: P values are just the tip of the iceberg by Jeffrey T. Leek & Roger D. Peng.
From the summary:
Ridding science of shoddy statistics will require scrutiny of every step, not merely the last one, say Jeffrey T. Leek and Roger D. Peng.
From the post:
Leek and Peng are right but I would shy away from ever claiming “…evidence-based data analysis.”
You can disclose the choices you make at every stage of the data pipeline but the result isn’t “…evidence-based data analysis.”
I say that because “…evidence-based data analysis” implies that whatever the result, human agency wasn’t a factor in it. On the contrary, an ineffable part of human judgement is a part of every data analysis.
The purpose of documenting the details of each step is to enable discussion and debate about the choices made in the process.
Just as I object to politicians wrapping themselves in national flags, I equally object to anyone wrapping themselves in “evidence/facts” as though they and only they possess them.