ChatGPT shows poor performance in answering drug-related questions

ChatGPT provided no response or incomplete or wrong answers to nearly three-quarters of drug-related questions reviewed by pharmacists, according to a study presented at the American Society of Health-System Pharmacists Midyear Clinical Meeting, held from Dec. 3 to 7 in Anaheim, California.

Sara Grossman, Pharm.D., from Long Island University in New York, and colleagues asked the publicly available, free version of ChatGPT 39 questions that had been posed to a drug information service. Two investigators compared ChatGPT responses to benchmark responses based on a literature search.

Of the 39 questions, the researchers reported that ChatGPT provided responses for 72 percent. Just over one-third of ChatGPT responses (36 percent) were accurate and complete and had no irrelevant information, whereas 64 percent had inaccurate or incomplete responses. One-third of the complex questions and 23 percent of the noncomplex questions were answered in a satisfactory manner.

"Health care professionals and patients should be cautious about using ChatGPT as an authoritative source for medication-related information," Grossman said in a statement. "Anyone who uses ChatGPT for medication-related information should verify the information using trusted sources."