JMIR Publications

ABSTRACT

Background:

Large language model agents capable of generating and executing statistical code from natural language may broaden access to clinical data analysis, yet which pipeline stages they perform reliably and which require expert oversight remains poorly defined.

Objective:

To evaluate the performance and systematic failure modes of an LLM agent across five stages of a clinical data analysis workflow.

Methods:

The publicly available dataset and R script from a previously published study of 12-year outcomes in 7,802 eyes with neovascular age-related macular degeneration at Moorfields Eye Hospital. Participants: An LLM agent (Claude, Anthropic) was evaluated in three interaction modes (Chat, Code, Cowork) is asked for three levels of data analysis practice: Prompt A is to generate research questions from raw data only; Prompt B is to develop a statistical analysis plan (SAP) from a high-level Clinical objective, then execute it; Prompt C is to execute an analysis given an investigator-drafted SAP. Each replicated three times (27 total runs). Qualitative evaluation of research question thematic coverage (Prompt A); SAP completeness against a reference checklist (Prompt B); evaluation of execution outputs against validated reference values and result text and narrative summary to execution logs (Levels B, C).

Results:

The agent generated 18 clinically grounded questions spanning seven domains; Cowork mode uniquely reached three thematic areas requiring data-driven methods. All nine SAPs correctly identified the statistical framework. Kaplan-Meier estimates were near-identical across 17 completed runs. Systematic execution errors emerged: SAP quality did not predict code correctness; and within-mode errors propagated identically across independent repetitions. Result text accurately reflected execution logs in nearly all runs, though unit propagation and an undisclosed post-crash re-run were identified. Of 17 narrative summaries, 8 were fully satisfactory; two runs produced clinically meaningful errors.

Conclusions:

LLM agents perform reliably for question generation and SAP drafting but require expert verification of formula composition, cohort boundary logic, and concordance computation before results are reported. Using an ophthalmology dataset as a controlled testbed, this study develops and applies an evaluation framework whose lessons are likely applicable across clinical specialties.