Creating medical exam questions with ChatGPT

For a recent study, UKB researchers created two sets of 25 multiple-choice questions (MCQs), each with five possible answers, one of which was correct. The first set of questions was written by an experienced medical lecturer; the second set was created by ChatGPT. 161 students answered all questions in random order. For each question, students also indicated whether they thought it was created by a human or by ChatGPT.

Matthias Laupichler, one of the study authors and research associate at the Institute for Medical Didactics at the UKB, explains, "We were surprised that the difficulty of human-generated and ChatGPT-generated questions was virtually identical. Even more surprising for us is that the students could not correctly identify the question's origin in almost half of the cases."

"Although the results obviously need to be replicated in further studies, the automated generation of exam questions using ChatGPT and co. appears to be a promising tool for medical studies."

His colleague and co-author of the study, Johanna Rother, adds, "Lecturers can use ChatGPT to generate ideas for exam questions, which are then checked and, if necessary, revised by the lecturers. In our opinion, however, students in particular benefit from the automated generation of medical practice questions, as it has long been known that self-testing one's own knowledge is very beneficial for learning."

Tobias Raupach, Director of the Institute of Medical Didactics, continues, "We knew from previous studies that language models such as ChatGPT can answer the questions in medical state examinations. We have now shown for the first time that the software can also be used to write new questions that hardly differ from those of experienced teachers."

Tizian Kaiser, who is studying human medicine in his seventh semester, comments, "When working on the mock exam, I was quite surprised at how difficult it was for me to tell the questions apart. My approach was to differentiate between the questions based on their length, the complexity of their sentence structure, and the difficulty of their content."

"But to be honest, in some situations, I simply had to guess, and the evaluation showed that I could barely differentiate between them. This leads me to the conviction that a meaningful knowledge query, as in this exam, is also possible exclusively through questions posed by the AI."

He is convinced that ChatGPT has great potential for student learning. It allows students to repeat what they have learned in different ways and in different ways again and again.

"There is the option of being quizzed by the AI on predefined topics, having mock exams designed, or simulating oral exams in writing. The repetition of the material is thus tailored to the exam concept, and the training possibilities are endless," says the study participant, while also qualifying, "However, I would only use Chat-GPT for this purpose and not beforehand in the learning process, in which the study topics have to be worked through and summarized."

"Because Chat-GPT is excellent for repetition, I fear errors can occur when preparing learning content. I wouldn't notice these errors without a prior overview of the topic."

It is known from other studies that regular testing—even and especially without grading—helps students to remember learning content more sustainably. Such tests can now be created with little effort. However, the current study should first be transferred to other contexts (e.g., other subjects, semesters, and countries), and it should be investigated whether ChatGPT can also write questions other than the multiple choice questions commonly used in medicine.

The research is published in Academic Medicine.