JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) have the potential to revolutionize ophthalmic care, but their performance evaluations remain fragmented. A systematic assessment is crucial to identify gaps and guide future research and clinical integration.

Objective:

This systematic review aims to characterize the current landscape of LLM performance evaluations in ophthalmology, focusing on model usage, data modalities, ophthalmic subspecialties, medical tasks, evaluation dimensions, and clinical alignment.

Methods:

A comprehensive search of PubMed, Web of Science, Embase, and IEEEXplore was conducted up to November 2024, identifying 817 unique publications. After screening, 165 peer-reviewed studies and preprints quantitatively evaluating existing LLMs in ophthalmic tasks were included. Data extraction, categorization, statistical analysis, and visualization were systematically performed with Python.

Results:

The review revealed a heavy reliance on closed-source LLMs (ChatGPT:98.18%, Gemini:34.55%, Copilot:19.39%), with minimal attention to open-source alternatives (LLaMA:3.03%). Modality-wise, 92.12% of studies focused on text-only tasks, while only 7.88% incorporated image-text evaluations, despite the centrality of imaging in ophthalmology. Subspecialty coverage was highly imbalanced, with comprehensive ophthalmology (35.76%), retina & vitreous (16.36%), and glaucoma (11.52%) dominating, while ocular pathology & oncology and ophthalmic pharmacology were largely neglected. Medical tasks primarily included medical queries (43.03%), standardized examinations (23.03%), and diagnosis formulation (13.94%), with limited exploration of triaging (4.24%) and disease prediction (2.42%). Accuracy (93.33%) was the predominant evaluation metric, while calibration and uncertainty were rarely addressed (2.42%). Clinically, real-world patient data usage (19.39%), non-English evaluations (4.85%), and in-clinic deployment (1.21%) remained critically understudied, highlighting a significant translational gap.

Conclusions:

This review highlights significant gaps in LLM evaluations, including uneven subspecialty coverage, limited multimodal assessments, and insufficient real-world clinical testing. Future research should prioritize standardized frameworks, unified benchmarks, and comprehensive real-world evaluations to ensure LLMs' safe and effective integration into ophthalmic practice, ultimately improving patient outcomes.