[Scientists] OSCAward for ALL-IN-META-BCG-CORONA

Here I share the story of the project that motivated me to nominate it, see the nomination text below. Lessons learned in terms of the unique features of the ALL-IN approach appeared in an updated (V2) version of the ALL-IN paper:

ter Schure J and Grünwald P. ALL-IN meta-analysis: breathing life into living systematic reviews and prospective meta-analyses [version 2; peer review: 1 approved, 2 approved with reservations]. F1000Research 2025, 11:549 (https://doi.org/10.12688/f1000research.74223.2)

OSCAward nomination

3. Project Information

a. Project title

ALL-IN-META-BCG-CORONA

b. Project category

Open access/ Open data/ Open materials/ Open software

4. Project Description

a. Objective of the project

Launched in the spring of 2020, this project’s first objective was to reach a conclusion on a COVID-19 related scientific question as soon as possible. When the scientific question became less relevant, its subsequent objective was to make all details available of a collaborative approach to clinical trial research that would exemplify (1) open science principles, (2) reducing research waste, (3) living materials that are fully citable and findable by citations, (4) rewarding all roles in the process by citations, (5) show-case statistical methods that facilitate collaboration like none other can, and (6) reach the audience that would like to repeat such a project (with a 2025 update to the ALL-IN methodological paper).

b. Project description

The ALL-IN approach to prospective meta-analysis promotes many open science principles. Hence the nomination of this project as the first application of the approach.

A second rationale for nomination is that this successful project looks like a failure: (1) Slow data sharing: even with everything (statistical design and dashboard infrastructure) in place to do efficient science, the interim results still could not affect the value/research waste of new trials. (2) A paper stuck in the preprint stage that (3) is hardly cited within the BCG research field, because (quote by a clinical researcher) ‘citing preprints is frowned upon in clinical research’, even when reviewed by 22 authors, in line with EQUATOR Network research reporting checklist, and openly available with a complete Replication Package. (4) A mostly failed attempt to use the underlying scientific work to give credit, make scientific evaluation findable, and highlight authors module-by-module.

The reason for giving this project more publicity is that even with knowledge of how it turned out, if I could go back in time, I would do everything exactly the same. It is still the best way to adhere to Open Science principles that I know about. (Although it would have been better if I already had the funding for legal-statistical research into sharing data without involvement of legal departments.) It started with excited collaborators, but was a lonely process after everyone lost interest in the scientific question (after rollout of COVID-19 specific vaccines (Pfizer/BioNtech, Moderna etc.)), and evolving scientific research pointed to futility.

5. Open Science Principles, Impact, and Sustainability

a. Alignment with open science values

Collaboration in a prospective meta-analysis avoids research integrity problems like filedrawering, outcome switching, incompatible outcome measures, unnecessary heterogeneity, small sample size, unnecessary new trials, and unnecessary delays in scientific conclusions if a living analysis is used. The latter is facilitated by the ALL-IN statistical approach that allows for bottom-up collaboration, in contrast to other statistical approaches to sequential analysis that require outside control over stopping the participating studies.

b. Potential to improve science or benefit society

The methodology reached the interest of the EU-RESPONSE consortium that added an ALL-IN work package into to their Horizon pandemic preparedness grant, based on the complete example of the methodology in the ALL-IN-META-BCG-CORONA project.

The experience with data transfer agreements inspired a research proposal that will study how to do such ALL-IN prospective meta-analysis based on sharing summary statistics alone (in a federated fashion) such that data sharing agreements might not be needed. This work package is part of a larger research proposal to adapt the ALL-IN methods developed for COVID-19 trials to trials in oncology. The proposal was evaluated as ‘fundable’ in the ZonMw Open Competition of 2023 and reached the interview round in 2024, and was resubmitted in the ZonMw Open Competition of 2025.

c. Potential to foster diversity and inclusion

Some countries have more difficulty recruiting participants in clinical trials than others. Hungary, for example (HU in the collaboration), experienced much more vaccine hesitancy than other countries and had low participation in vaccine trials because of it (and hence small sample size). Not only did this collaboration save this data from the filedrawer, it also put the research team on equal footing with the others due to the bottom-up nature of the statistical approach that is agnostic to sample size.

The ResearchEquals platform was used for the replication package, not only to make the project reproducible and replicable but to achieve three additional goals: First, to give credit to other important work of those not involved in the ALL-IN-META-BCG-CORONA project as authors, but indispensable for the trial design and data collection. Second, to make standalone modules findable through citations that have their own scientific value. And third to highlight the different authorships of the various modules.

d. Sustainability

ResearchEquals might not be a sustainable platform (relatively new), so there was a trade-off with the three additional goals mentioned above. Software is available in an R package on CRAN, within the EU-RESPONSE consortium a grant is available to hire a software specialist to create sustainable software.

6. Public Summary of the Project

a. Title

ALL-IN-META-BCG-CORONA: open science methods showcased on ResearchEquals

b. Public Summary

ALL-IN-META-BCG-CORONA is a prospective living meta-analysis of clinical trials that studied the protection of the BCG vaccine (originally developed to protect against tuberculosis) against COVID-19 infections and hospitalizations. The statistical methods in the project add to other methods for prospective meta-analysis by their bottom-up nature that puts all participating clinical trial teams on an equal footing in the collaboration. The resulting paper is openly available and has a replication package that not only inspires similar projects and allows for reuse of documentation, but also aimed for goals to make standalone supplementary material findable through citations, highlight authorship and give credit to authors that for logistical reasons could not author the publication. While the project failed those latter goals because platforms like Google Scholar, PubMed, Scopus stopped showing/never showed ResearchEquals citations, the ideas behind the attempt are worth sharing within the open science community.

MedRxiv preprint ALL-IN-META-BCG-CORONA

Bacillus Calmette-Guérin vaccine to reduce COVID-19 infections and hospitalisations in healthcare workers – a living systematic review and prospective ALL-IN meta-analysis of individual participant data from randomised controlled trials

J.A. (Judith) ter Schure, Alexander Ly, Lisa Belin, Christine S. Benn, Marc J.M. Bonten, Jeffrey D. Cirillo, Johanna A.A. Damen, Inês Fronteira, Kelly D. Hendriks, Ana Paula Junqueira-Kipnis, André Kipnis, Odile Launay, Jose Euberto Mendez-Reyes, Judit Moldvay, Mihai G. Netea, Sebastian Nielsen, Caryn M. Upton, Gerben van den Hoogen, Jesper M. Weehuizen, Peter D. Grünwald, C.H. (Henri) van Werkhoven medRxiv 2022.12.15.22283474; doi: https://doi.org/10.1101/2022.12.15.22283474

ResearchEquals replication package ALL-IN-META-BCG-CORONA

https://doi.org/10.53962/kyep-h9

8. References

Blog post detailing the excitement of doing prospective meta-analys in a live dashboard and initiating the ALL-IN-META-BCG-CORONA project: https://significanthelp.nl/blog/Personal__COVID-19_trial_dashboard

Media coverage of the innovative statistical approach: https://significanthelp.nl/blog/General__Interview_in_Het_Parool_en_New_Scientist

Blog post detailing the failure of fast data sharing, and how the research effort could have been more valuable without data transfer agreements: https://significanthelp.nl/blog/Scientists__Avoidable_research_waste_caused_by_data_transfer_agreements

Blog post detailing the failure of using citations of underlying scientific effort to give credit, make modules findable, and highlight authorship: https://significanthelp.nl/blog/Scientists__What_I_tried_to_achieve_with_a_ResearchEquals_collection

Journal publication about the benefits of the ALL-IN statistical approach for bottom-up collaboration updated into a V2 with the ALL-IN-META-BCG-CORONA results: