Parallel Session at the Tilburg Meta-Research Day
Below the abstract and my take-away of the discussion.
How can meta-research improve statistical research/practice?
Meta-research has spurred two opposing perspectives on science: one treats individual studies and study series as two separate publication types (e.g., in Ioannidis' 2005 paper 'Why Most Published Research Findings Are False') and one treats individual studies as part of a series that needs to be informed by that series (e.g., in the Chalmers et al. 2014 paper 'How to increase value and reduce waste when research priorities are set'). These perspectives decide what statistical methods are valid, both at the individual-study level and at the meta-analysis level. Could meta-research also inform which perspective is most appropriate and improve the corresponding statistical research and practice?
It varies a lot per field whether scientists in their experimental design actually feel like they contribute to an accumulating series of studies. In some fields awareness exists that the results of an experiment will someday end up in a meta-analysis with existing experiments, while in others scientists aim to design experiments as 'refreshingly new' as possible. In a table that shows series of studies together in one column if they could be meta-analyzed, this latter approach shows scientists who mainly aim to initiate new columns. This pre-experimental perspective might be different from the meta-analysis perspective, in which a systematic search and inclusion criteria might still force those experiments together in one column, even though they weren't intended that way. This practice might erode trust in meta-analyses that try to synthesize effects from too different experiments.
The discussion was very hesitant towards enforcing rules (e.g. by funders or universities) on scientists in priority setting, such as whether a field needs more columns of 'refreshingly new' experiments, or needs replications of existing studies (extra rows) so a field can settle on a specific topic in one column with a meta-analysis.
In terms of statistical consequences, sequential processes might still be at play if scientists designing experiments know about the results of other experiments that might end up in the same meta-analysis. Full exchangeability in meta-analysis means that no-one would have decided differently on the feasibility or design of an experiment had the results of others been different. If that assumption cannot be met, we should consider studies as part of series in our statistical meta-analysis, even without forcing this approach in the design phase.
For a full summary of the discussion, see this document:
Judith ter Schure - Tilburg Meta-Research Day Parallel Session Statistics summary.pdf
Mon, 18 November