[Peers/Scientists] Accumulation Bias: How to handle it ALL-IN

Yesterday, members of the VVSOR received the latest issue of our society magazine STAtOR with the seven-page article above. Credits for the nice format go to the STAtOR copy-editor Monique van Hootegem and the STAtOR editorial board that encouraged me to rewrite this piece. The original article appeared as a blogpost on The Replication Network thanks to the very kind invitation of Professor Bob Reed.

The article argues for ALL-IN meta-analysis, Anytime Live and Leading INterim meta-analysis. It comes with a perspective on science that does not treat a series of studies as a fixed batch, but as a growing process that allows decisions to start or stop the research effort for which the meta-analysis itself could be Leading.

The article first introduces the concept of Research Waste and the need to better inform new research by past results. Yet to achieve that ideal, we need to move away from conventional meta-analysis methods, since they suffer from Accumulation Bias if data-driven decisions are made on whether new research is necessary or wasteful. The selection of which studies to replicate affects the sampling distribution underlying conventional statistical methods. ALL-IN meta-analysis handles that problem by shifting the focus from a single sample distribution (that needs to condition on the sample and study series size), to a growing series.

Here you can find the STAtOR article:

STAtOR 2021 1, 21-27 JterSchure LR.pdf

The references and technical details can be found in a longer version of the article that is available in the CWI repository here.
The R code that runs the simulations and produces the plots can be found here: Code Accumulation Bias How to handle it ALL-IN.R