Office of Medical Communications

Christopher Brady, M.D., M.H.S., associate professor of surgery

(AUGUST 23, 2024) Christopher Brady, M.D., M.H.S., associate professor of surgery, commented to Healio about his research on the use of large language models (LLMs) in generating scientific abstracts.

There is a lot of hype around the use of large language models (LLMs), Brady said, especially in medicine.

Brady and colleagues conducted a study to determine if a LLM could generate an accurate abstract if it were given the full text of a scientific research article. Brady said this allowed the comparison of accuracy in LLMs vs. author-written abstracts. The work was presented at the American Society of Retina Specialists annual meeting.

“At one extreme, there is a hope that one day LLMs may be able to process an entire medical chart and generate an unconsidered hidden diagnosis or a novel therapeutic strategy that the team had not considered yet,” he said. “That being said, the capacity of these systems to hallucinate or generate completely preposterous information is well documented.”

A published paper on the OAKS and DERBY trials without the abstract was input into Google Bard, a free version of ChatGPT 3.5 and the paid version of ChatGPT 4. Based on inaccurate abstracts generated by Bard and ChatGPT 3.5, ChatGPT 4 was chosen to be used for the rest of the study.

The biggest limitation is that AI systems keep changing; since the study was conducted in January, Bard no longer exists and is now Gemini, and while ChatGPT 4 still exists, ChatGPT 4o is the most advanced model, “so these results can change with each additional enhancement,” Brady said.

“We were impressed that ChatGPT 4 was able to process these manuscripts and generate a uniform scientific abstract that looked normal and did not have mistakes,” he said. “We feel that these could immediately prove to be useful tools for authors, peer reviewers and editors to make articles more consistent and more correct.”