JMIR Publications

ABSTRACT

Background:

Large language models (LLM) have the potential to enhance clinical flow and improve medical education, but they encounter challenges in specialized knowledge in ophthalmology.

Objective:

To enhance ophthalmic knowledge by refining a general LLM into an ophthalmology-specialized assistant for patient inquiries and medical education.

Methods:

We transformed the Llama2 into an ophthalmology-specialized LLM, termed EyeGPT, through three strategies: prompt engineering for role-playing, finetuning with publicly available datasets filtered for eye-specific terminology (83,919 samples), and retrieval-augmented generation leveraging a medical database and 14 ophthalmology textbooks. Four board-certified ophthalmologists evaluated the efficacy of various EyeGPT variants through a comprehensive evaluation of using 120 diverse category questions in both simple and complex question-answering scenarios. The performance of the best EyeGPT model was then compared to that of the unassisted human physician group and the EyeGPT+human group. Four metrics were proposed for assessment: accuracy, understandability, trustworthiness, and empathy. The proportion of hallucinations was also reported.

Results:

The best-finetuned model significantly outperformed the original Llama2 model in providing informed advice (9.30 ± 4.42 vs. 13.79 ± 5.70, P<.001) and mitigating hallucinations (80.8% vs. 44.2%, P<.001). Incorporating information retrieval from reliable sources, particularly ophthalmology textbooks, further improved the model's response compared to the solely best-finetuned model (13.08 ± 5.43 vs. 15.14 ± 4.64, P=.001), and reduced hallucinations (59.2% vs. 47.4%, P=.02). Subgroup analysis revealed that EyeGPT showed robustness across common diseases, with consistent performance across different users and domains. Among the variants, the model integrating finetuning and book retrieval ranked highest, closely followed by the combination of finetuning and manual database, standalone finetuning, and pure role-playing methods. EyeGPT demonstrates competitive capabilities in understandability and empathy when compared to human ophthalmologists. With the assistance of EyeGPT, the performance of the ophthalmologist is notably enhanced.

Conclusions:

We pioneering introduced EyeGPT by refining a general domain LLM, and conducted a comprehensive comparison and evaluation of different strategies to develop an ophthalmology-specific assistant. Our results highlight EyeGPT's potential to assist ophthalmologists and patients in medical settings.