New method developed to improve accuracy of disease diagnosis

A recently released study in the journal Statistics in Medicine describes a new method, named the Observers Needed for Evaluation of Subjective Tests (ONEST), developed to determine the optimal number of pathologists needed for a correct diagnosis. The method was developed by Gang Han, Ph.D., associate professor in the Department of Epidemiology and Biostatistics at the Texas A&M University School of Public Health, with Bohong Guo, a former Master of Public Health student who majored in biostatistics (2019-2021) and colleagues from the Moffitt Cancer Center & Research Institute, Saint Louis University School of Medicine, and Yale University School of Medicine.

Han and colleagues developed a statistical framework to assess the performance of a diagnostic test with multiple observers. The proposed method includes an exploratory analysis, a statistical test of whether the observers' agreement percentage will plateau to a non-zero value, and a statistical model to estimate the agreement percentage and the number of observers for reaching the plateau.

This method was applied in a non-small cell lung cancer example and a triple negative breast cancer example using reads of the immunohistochemical tests with SP142 and SP263 assays for expression of Programmed death-ligand 1 (PD-L1) to determine the number of observers needed for evaluation of the subjective tests.

The proposed method can indicate whether adding more observers to a test causes the proportion of agreement to plateau. Cases where the curve does not plateau could indicate an unreliable test. In cases where the curve does flatten, the method indicates at least how many observers are needed to reach a stable and reliable estimation of their agreement.

Better understanding of how many observers are needed for optimal accuracy on a diagnostic test will help improve correct diagnosis, the right level of care and disease treatment. The authors believe this method could be utilized by test creators and regulatory agencies to evaluate newly proposed subjective laboratory tests at different numbers of pathologists, which can ensure that the test will perform reliably in real-world settings.