What's the best way to rate the quality and quantity of research produced by orthopaedic surgery residency programs? A new "research impact ranking" provides a more objective approach to assessing and comparing research productivity at US orthopaedic departments, according to a study in the November Journal of Orthopaedic Trauma (JOT).
Partly motivated by the desire to track the success of their own program's research effort, Dr. Louis B. Jones and colleagues of University of Mississippi Medical Center, Jackson, developed an alternative approach to ranking orthopaedic surgery residency training programs in terms of research productivity. The researchers write, "We have created a reproducible and meaningful residency ranking system that will allow programs to evaluate their progress both against their peers and within their own systems over time."
System Ranks Quality as Well as Quantity of Research by Orthopaedic Surgery Trainees
Research, along with clinical care and academic performance, is a key part of the mission of orthopaedic surgery training programs. But research productivity is difficult to measure objectively. Simply counting the number of studies published accounts for the quantity but not necessarily the quality of research produced.
Dr. Jones and colleagues sought to develop a benchmarking system, or "h-index," for residency programs. From a list of 72 orthopaedic surgery journals, they identified 48 "relevant, impactful" journals based on the Impact Factor, a traditional measure of the importance of medical journals; and a newer alternative metric called the Eigenfactor. The journals with the most citations from US residency programs were Journal of Bone and Joint Surgery (American edition), Spine, and Clinical Orthopaedics and Related Research.
Dr. Jones and colleagues then created a database of nearly 103,000 articles published in these journals from 2005 to 2015. These articles were then attributed to the 157 US orthopaedic surgery residency programs. The researchers put their model through several iterations to identify the most effective search process.
The results provided a ranking of the orthopaedic training programs with the most productive research efforts, based on the number of citations in high-quality journals. The article in Journal of Orthopedic Trauma presents the full list of programs and their rankings. The data can also be accessed at the University of Mississippi Medical Center Department of Orthopaedic Surgery and Rehabilitation research page.
Because the ranking system accounted for the number of articles published, larger programs—based on number of trainees and number of faculty—tended to have higher rankings. Each of the six top-ranked programs had more than 1,000 research publications and more than 20,000 citations over the ten-year period studied.
The rankings in terms of number of citations were closely correlated with the number of articles published in the select orthopaedic surgery journals. The rankings were "moderately" correlated with other systems for ranking training programs.
Dr. Jones and colleagues note some limitations of their ranking system—notably including the fact that it's sometimes difficult to attribute articles to a specific residency program.
Nevertheless, they believe their ranking system, based on "clear and reproducible" methods, provides a useful approach to assessing and comparing the research productivity of orthopaedic residency programs. The authors conclude, "It is relevant and based on measurable data that is the cornerstone of an academic surgeon's practice: publishing articles to contribute to the orthopaedic knowledge base."
Explore further: Authors with financial conflicts reporting negative outcomes in major orthopaedic journals
Louis B. Jones et al. An Objective Methodology to Assess Meaningful Research Productivity by Orthopaedic Residency Departments, Journal of Orthopaedic Trauma (2018). DOI: 10.1097/BOT.0000000000001106