[Scientists] What I tried to achieve with a ResearchEquals collection
How I did the best I could and spent a large amount of time, still gave up before fully completing it, am happy I tried it, happy I gave up, and happy with the result.
ResearchEquals is a research publishing platform that is used by open science enthusiasts like myself. This blog post is about my use of the platform for the Replication Package ALL-IN-META-BCG-CORONA.
You can find it here: Replication Package ALL-IN-META-BCG-CORONA
This collection transparently details a collaboration with fourteen clinical researchers representing seven clinical trials (NL, SA, US, DK, HU, BR, AF) – in a prospective meta-analysis with three senior researchers in a steering committee, two statisticians, and an external risk-of-bias team.
The ResearchEquals collection tries to make all supplementary material of the collaboration available for anyone that wants to reproduce our work or start a similar project, even though much of it was already available on our project website (now out-of-date). Here I want to describe why I chose the ResearchEquals platform for the complete collection: while many other platforms could also serve to make the project reproducible, choosing ResearchEquals had to do with additional goals.
Three additional goals…
First, to give credit to other important work of those not involved in the ALL-IN-META-BCG-CORONA project as authors, but indispensable for the trial design and data collection. Second, to make standalone modules that have their own scientific value findable through citations. And third, to highlight the different authorships of the various modules in this project.
To achieve these goals I collaborated with ResearchEquals creator Chris Hartgerink to make two step-by-step guides for my collaborators so we would optimally leverage the possibilities of the platform. I’ve attached these guides to the ALL-IN-META-BCG-CORONA collection to share the idea – see the step-by-step guides here on ResearchEquals: (https://www.researchequals.com/modules/wgtb-cagt)[https://www.researchequals.com/modules/wgtb-cagt] – but unfortunately, my goals…
… were not really achieved
Only the third goal could be partly achieved, but not for a lack of trying. The main problem was that third parties did not integrate the citation data generated by ResearchEquals. Partly because of that, I failed to motivate other authors to make every protocol and risk-of-bias assessment available, and gave up on fully achieving the three additional goals, described in detail below. Yet I am still happy that I tried this and happy with the result: the Replication Package gives complete supplementary material to reproduce our analysis, and is very nicely presented on the ResearchEquals platform.
With this story I hope to highlight to other open science enthusiasts that these goals are still worth pursuing, but a lot more difficult than I thought.
Additional goal 1: Give credit to important other work
ResearchEquals generates structured citation data that I tried to use to give credit to other important work. The Statistical Analysis Plan of the project, for example, is heavily inspired by the protocol publication of the first clinical trial (the NL one), with many researchers involved that were not authors of our project. By using ResearchEquals I could not only give our Statistical Analysis Plan a DOI and make it available, but also add a citation to the NL protocol. Other platforms to make such documents available, like OSF, do not allow to add a citation in the same way, but only in a reference section within a pdf. References within a pdf on a platform like OSF are not found by any citation data platform, not even by Google Scholar or Scopus Preprint search.
Giving credit to all the researchers involved in the trials was important because of the logistics of a prospective meta-analysis that allowed only two authors per trial: a Principle Investigator and a Data Uploader. By generating additional citations to their protocols and motivating them to also publish their data such that I could cite it, I intended to give credit to all important researchers involved in generating the data for the meta-analysis. However, motivating others to generate DOIs for their protocols and data takes time. The flexibility of the ResearchEquals collection would allow us to generate citations of new DOIs also after the paper was published (when the publication itself could not cite this supplementary material anymore), by adding supplementary documents to the collection that could generate new citations.
Additional goal 2: Make ‘standalone’ modules findable
In a prospective meta-analysis, decisions on study inclusion are made with no trial results available. So risk-of-bias assessments were performed based on the protocols of the participating randomized controlled trials.
Some of these protocols were published (NL, DK, BR) and so their DOIs could be added to the collection. ResearchEquals makes them findable as supplementary material to our project, but because they are published in journals, they are already easily findable trough scholarly data platforms like Google Scholar or PubMed. The risk-of-bias assessments of these trial protocols therefore also have standalone scientific value to anyone that comes across these protocols outside of our supplementary material. The way to lead a reader of these protocol publications to the risk-of-bias assessments would be the ‘Cited by’ functionality on a journal website or within a scholarly data platform.
ResearchEquals assigns a DOI to our risk-of-bias assessments, and generates citation data for crossref. You can see in this example risk-of-bias assessment of the BR trial protocol that it cites the protocol publication: https://www.researchequals.com/modules/r30w-ksmt. However, these citations are currently not available anywhere other than ResearchEquals itself.
Third-party failures
There was a point at which these citations were findable through Scopus. But Scopus was the only one, and instead of this situation improving over time, it became worse since also Scopus does not display ResearchEquals citations anymore. For example, the citation by the risk-of-bias assessment of the BR trial publication (link above) is not findable through the ‘Cited by’ functionality in any of the six mode widely used ones:
Publisher’s (BMC/Springer Nature) Metrics page of the Trials protocol publication
Web of Science ‘Cited By’ section
Google Scholar ‘Cited By’ section
Google Scholar and the Scopus Preprints-‘Cited by’ functionality (in Beta version) do find the citation of the protocol by the MedRxiv preprint. So in that way, I partly achieved to give credit to the three published protocols. But these citations can only indirectly lead a reader to the risk-of-bias assessments of these protocols via the MedRxiv preprint that cites the Replication Package, and therefore do not highlight those as standalone modules with scientific value.
For protocols that we intended to make available after the preprint publications the process turned out to be completely ineffective to highlight their risk-of-bias assessments: citations from ResearchEquals would not be findable, so it was ineffective to give credit to the protocol authors (Additional goal 1) as well as lead readers of the protocol to the risk-of-bias assessment (Additional goal 2). This was demotivating, and the interdependencies between the modules made the process of achieving anything with ResearchEquals citations very time-consuming (see below: Interdependencies and waiting).
Additional goal 3: Highlighting authorship of modules
Not all authors were involved in all parts of the project. For example, an external team performed the risk-of-bias assessments that was not involved in any trial, but worked with a representative of Cochrane Netherlands. By making their risk-of-bias assessments available as modules in ResearchEquals, I could show who the authors were and directly connect these modules to their research output by feeding into their ORCID profiles.
However, given that I am myself author of many of these modules, I know that they can easily clutter your ORCID profile. I decided to hide most of them from my public profile, and also informed the other authors that they could do that in the step-by-step guide. Of course, this makes the effort to align authorship with modules less effective. Maybe if the reward structures change, showing this type of research output will be beneficial for researchers in the future and this feeling of ‘cluttering’ your profile will cease to exist. I am proud of the Code to reproduce the analysis and figures, and happy of the automatic feeding into my ORCID profile: https://orcid.org/0000-0002-2147-5510 But unfortunately, the automatic process classifies it as a ‘Book chapter’ and also does not communicate with Scopus, such that it does not automatically appear on the Pure profile my employer wants me to use: https://amsterdamumc.org/en/research/researchers/judith-ter-schure-1.htm
Interdependencies and waiting…
To achieve all the goals I had on the ResearchEquals platform, I had to motivate others to create profiles and make sure references were added correctly because you can only publish a module once. This introduced a lot of dependencies between the modules and only one order in which they could be published and linked together by citations. Authors had to wait for others to publish modules so very little was published…
The major complicating factor was that trial protocols are often amended, so the latest version that was assessed for risk-of-bias might not be the same version that was published. For a fair representation of the process that led to the risk-of-bias assessment I implemented a check on the version our risk-of-bias team received whether that was the same protocol that was published, and if not, ask the clinical trial researchers to explain in a few sentences what the changes were and add that to the risk-of-bias module on ResearchEquals. Hence this process had many steps and many people involved to do well: First, ask the research team whether their protocol is published, and if it is not, whether they want to post it as a module on Research Equals under their own authorship. Wait for them to do that. Second, ask them whether that version of the protocol is the one that was assessed for risk-of-bias and if not, provide detail. Third, ask the risk-of-bias team to wait for this input before posting their assessment on Research Equals such that the citation of the protocol has sufficient context.
Giving up
This entire scheme turned out to require a large amount of reminders to motivate the other authors, even with a step-by-step guide available. See the step-by-step guides here on ResearchEquals: https://www.researchequals.com/modules/wgtb-cagt
So it culminated in only one available risk-of-bias assessment of a published protocol that was exactly the same as the one assessed: the BR protocol. This research team of the BR (Brazilian) trial did actually appreciate the citation of their protocol that, thanks to the ResearchEquals platform, was fed into Crossref at the time. But, as we saw above, this citation is not findable anymore.
Unfortunately, the trial teams without protocol publications were not immediately posting their protocols on ResearchEquals, because they were still considering publishing them in a journal. Therefore, the risk-of-bias team had to wait posting the risk-of-bias assessments, to make sure these could cite the protocols. And the waiting resulted in postponing, which resulted in giving up.
The lack of impact of these citations made me realize that in terms of completeness, probably no-one would bother to ever look at these risk-of-bias analyses as they were not findable as standalone modules. So they might not be worth all this effort.
With one risk-of-bias assessment available in the Replicabilion package, the replication package already transparently showed how these assessments were carried out. So I decided to stop bothering everyone. For their credit: the risk-of-bias team did allow me to add all the assessments on ResearchEquals (even if to do it quick, it would be under my authorship), but I did not want to do that without also making the protocols available that were not mine.
What is there
All the stuff you would expect in a replication package, like (summary) data, code, figures, Statistical Analysis Plan, instructions for data processing and upload etc, even the Newsletters that were sent during the pandemic. Published NL trial protocol Published DK trial protocol Published BR trial protocol BR trial protocol riskof-bias assessment
What is still missing
NL trial protocol riskof-bias assessment DK trial protocol riskof-bias assessment SA trial protocol SA trial protocol riskof-bias assessment US trial protocol US trial protocol riskof-bias assessment HU trial protocol HU trial protocol riskof-bias assessment AF trial protocol AF trial protocol riskof-bias assessment
Why I am happy anyway
As a replication package, I am happy with what is available on ResearchEquals. It is unfortunate that on the collection page, mostly my name as the ‘editor’ of the collection is visible and not the authors of the modules. Bu you can find their names by clicking and accessing the modules individually. The collection functionality puts everything nicely together and has its own DOI, which we linked in the MedRxiv preprint. In that way it was (and is) still possible to update this supplementary material. This is a functionality that we used extensively by filling most of the collection after the MedRxiv publication.
Even though I did not manage to meet most of my three additional goals above, I’m still happy that I tried. These goals are valuable enough to spend the time. But you can probably guess my level of frustration at some point. Now, that frustration is a long time ago and I feel that the effort might serve a purpose in the future. But of course, for that to even be possible, I have to openly share the story. So here it is.