JMIR Publications

ABSTRACT

Background:

Specialty triage at first contact is an overlooked step in early diagnostic pathways for rare diseases. Patients often present with overlapping, multisystem, and atypical manifestations, making first-visit specialty selection challenging and potentially prolonging diagnostic pathways.

Objective:

To evaluate the accuracy, response time, and consistency of large language models (LLMs) for initial-visit specialty triage in rare diseases across multiple datasets, and to compare their performance with registered nurses and non-medical participants.

Methods:

In this retrospective benchmarking study, we used five rare disease datasets: a publication-derived case set, three RareBench-derived datasets, and a Facial phenotype-Gene-Disease Dataset-derived set. Fourteen LLMs were evaluated over five independent runs per case. Performance was assessed using accuracy, response time, and consistency, with subgroup analyses by model accessibility, reasoning mode, parameter scale, and phenotype count. Human comparison was conducted on the publication-derived case set using registered nurses and non-medical participants.

Results:

Across datasets, accuracy varied from 0.4378 to 0.7141. Claude-opus-4-5 achieved the highest accuracy (0.7141) and consistency (0.9653), averaging 10.79 s per case. GPT-5.1 had the shortest response time (3.39 s per case) and high accuracy (0.6948). Proprietary models achieved higher average accuracy than open-weight models (0.6973 vs 0.6365). Standard models achieved higher average accuracy than thinking models (0.6789 vs 0.5826) and had shorter response times. Accuracy varied by phenotype count, with higher performance in cases with 1–2 or more than 14 phenotypes. On the publication-derived case set, LLMs achieved higher average accuracy than registered nurses and non-medical participants (0.5978 vs 0.4914 and 0.4573).

Conclusions:

LLMs showed potential as assistive tools for initial-visit specialty triage in rare diseases. Model choice, reasoning mode, and phenotype information density substantially influenced performance, suggesting that deployment should prioritize validated models with strong accuracy–efficiency balance. Future work should evaluate LLM-based specialty triage in prospective clinical settings and develop clinician-supervised workflows with traceable evidence support. Clinical Trial: Not applicable.