Expert panel diagnosis for diagnostic test poorly described, experts not blinded to test under study

Evaluation of diagnostic studies is often a challenge in diseases that are not defined by a specific test. Assessment of the accuracy of diagnostic tests is essential because they may be used to define who is considered to have a disease and receive treatment for it. However, measuring the accuracy of a diagnostic test requires an accurate gold standard, which defines which patients truly have and do not have the disease.

Studies of diseases not defined by a specific test often rely on expert panels to establish the gold standard. In a systematic review and analysis of the diagnostic literature using expert panels to define the gold standard for a given disease, Loes Bertens and colleagues from University Medical Center Utrecht determined how expert panels were used in such studies and how well their process was described and reliability assessed.

The authors evaluated 81 diagnostic studies published up to May 31, 2012, including studies of diagnostic tests for psychiatric disorders (30 of 81 papers, 37%), half of which pertained to dementia, cardiovascular diseases (17 papers, 21%), and respiratory disorders (10 papers, 12%). They found that reporting was often incomplete, with 83% of studies missing at least some important information about the expert panel. In 75% of studies the panel consisted of three or fewer members, and panel members were blinded to the results of the test results being evaluated in only 31% of studies. Blinding is important because knowledge of the index text results could influence the panelists' decision as to whether the patient had the disease. Reproducibility of the decision process was assessed in only 21% of studies.

The authors state, "Complete and accurate reporting is a prerequisite for judging potential bias in a study and for allowing readers to apply the same study methods. In total, only 14 (17%) papers reported complete data on key issues such as the panel constitution, the information presented to the panel and the exact decision process to determine the final diagnosis." They also found that despite publication of reporting guidelines, the completeness of reporting did not improve over time, perhaps because the reporting guidelines do not include specific criteria for expert panel diagnoses. The authors make a number of recommendations to improve reporting of expert panel diagnosis. They conclude, "Our review revealed a large variation in applied methods as well as major deficiencies in the reporting of key features of the panel diagnosis process…The results of our review may serve as a starting point in the development of formal guidelines on methodology and reporting of panel diagnosis."