Rigor and transparency index: Large-scale analysis of scientific reporting quality

researchers
Credit: CC0 Public Domain

JMIR Publications recently published "Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality" in the Journal of Medical Internet Research (JMIR), which reported that improving rigor and transparency measures should lead to improvements in reproducibility across the scientific literature, but assessing measures of transparency tends to be very difficult if performed manually by reviewers.

The overall aim of this study is to establish a scientific reporting quality metric that can be used across institutions and countries, as well as to highlight the need for high-quality reporting to ensure replicability within biomedicine, making use of manuscripts from the Reproducibility Project: Cancer Biology.

The authors address an enhancement of the previously introduced Rigor and Transparency Index (RTI), which attempts to automatically assess the rigor and transparency of journals, institutions, and countries using manuscripts scored on criteria found in reproducibility guidelines (e.g., NIH, MDAR, ARRIVE).

Using work by the Reproducibility Project: Cancer Biology, the authors were able to determine that replication studies scored significantly higher than the original papers, which according to the project, all required additional information from authors to begin replication efforts.

Unfortunately, RTI measures for journals, institutions, and countries all currently score lower than the replication study average. If they take the RTI of these replication studies as a target for future manuscripts, more work will be needed to ensure the average manuscript contains sufficient information for replication attempts.

Dr. Anita Bandrowski from the University of California San Diego said, "Research reproducibility is necessary for scientific progress. However, over the last decade, numerous reports on research irreproducibility have shed light on a lingering problem, one that is proving to be both troublesome and costly."

"Research reproducibility is necessary for scientific progress. However, over the last decade, numerous reports on research irreproducibility have shed light on a lingering problem, one that is proving to be both troublesome and costly."

In an effort to encourage reproducibility, numerous scientific organizations and journals have adopted the Transparency and Openness Promotion guidelines, which focus on establishing at the level of individual journals.

Along a similar vein, the publisher-driven Materials Design, Analysis, and Reporting framework is a multidisciplinary research framework designed to improve reporting transparency across life science research at the level of individual manuscripts.

This framework provides a consistent, minimum reporting checklist whose criteria were used, in part, to create the first RTI, a quality metric focusing on research methodologies and reporting transparency.

Specifically, the authors here introduce the latest version of the RTI, which represents the mean SciScore over a subset of papers, and demonstrate how it can be used to assess reporting transparency within research institutions.

While it is not practical to simply describe all papers scoring a "2" as not replicable and all papers scoring an "8" as replicable, as numerous fields and their subsequent best practices exist, it can be stated that higher scores are associated with more methodological detail and as such are likely easier to use to attempt a replication.


Explore further

Research result reporting set for boost under new system

More information: Joe Menke et al, Establishing Institutional Scores With the Rigor and Transparency Index: Large-scale Analysis of Scientific Reporting Quality, Journal of Medical Internet Research (2022). DOI: 10.2196/37324
Provided by JMIR Publications
Citation: Rigor and transparency index: Large-scale analysis of scientific reporting quality (2022, July 26) retrieved 2 October 2022 from https://medicalxpress.com/news/2022-07-rigor-transparency-index-large-scale-analysis.html
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.
2 shares

Feedback to editors