How accurate is ChatGPT at rating common allergy myths? Pretty accurate, says research

Artificial intelligence language models such as ChatGPT are increasingly being used in many different professions, and medicine is not an exception.

In a new study being presented at the American College of Allergy, Asthma and Immunology (ACAAI) Annual Scientific Meeting in Anaheim, Calif., ChatGPT was asked to rate 10 common allergy myths, and it judged their accuracy correctly 91% of the time.

"We searched the PubMed and Google Scholar databases to identify all articles on common allergy myths between 2000-2023," says Ricardo Estrada-Mendizabal, MD, ACAAI member and lead author of the study.

"Our goal was to assess the accuracy of ChatGPT in correctly identifying common allergy myths. We know allergy and asthma patients are constantly exposed to misinformation on the internet. We thought that the surge of public language models could represent a bigger threat to misinformation spread, especially because ChatGPT is unable to provide references."

Ten myths were selected and entered in ChatGPT 4.0 with the prompt "true or false." ChatGPT answers were graded by 24 allergists on a Likert scale of 1–4 (1=inaccurate [100% not true], 2=somewhat inaccurate [>30% not true], 3= somewhat accurate [<10% not true], 4=accurate [100% true]).

Finally, allergists were asked their likelihood of using ChatGPT for patient education with a Likert scale of 1–5 (1=not likely at all to 5=very likely). Overall, ChatGPT had an accurate or somewhat accurate response in 91% of cases and an inaccurate or somewhat inaccurate response in 9%.

Said Dr. Estrada-Mendizabal, "The response to whether a positive allergy test indicates a clinical allergy was the most accurate response (96% accurate), whereas the response to if hypoallergenic animals were better for people with asthma and allergies was the most inaccurate (17% inaccurate and 21% somewhat inaccurate response). Eighteen allergists (75%) were open (likely or very likely) to using ChatGPT in the future for patient education."

The authors of the paper say more studies are required to interpret the validity of ChatGPT in providing accurate medical information.