JMIR Publications

ABSTRACT

Background:

Specialty-relevant risk factors often appear during routine pediatric intake, but they are not always carried forward into age-appropriate action. Developmental dysplasia of the hip (DDH) is a useful test case because the risk signal may be clear in the history, even when the visit is dominated by fever, feeding, jaundice, rash, respiratory symptoms, or parental reassurance.

Objective:

We evaluated whether commercially available large language models (LLMs), used as a structured intake layer, could preserve previously provided DDH-relevant history and map it to an expert-defined next step more consistently than unaided community clinicians in a fixed-information vignette simulation.

Methods:

We created 48 standardized early-infant intake vignettes covering five DDH triage categories: no DDH-specific action, continued surveillance, hip ultrasound, pelvic radiograph, and urgent pediatric orthopedic referral. A blinded pediatric orthopedic reference panel established the triage standard. Twenty community primary care clinicians completed 2660 of 2880 planned audio-vignette assessments. Three LLMs (ChatGPT GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro) assessed verified transcripts in 20 independent reset sessions per model (2880 assessments). The primary outcome was exact agreement with the five-category expert standard. Secondary outcomes included action-required classification, surveillance/no-action classification, ordinal under-triage and over-triage, reliability, and LLM safety review.

Results:

Exact agreement was higher for LLMs than for clinicians (91.8%, 2644/2880 vs 74.1%, 1971/2660; difference 17.7 percentage points, 95% CI 15.1-20.3; P<.001). For action-required scenarios, LLMs agreed with the expert standard in 95.6% (1835/1920) of assessments, compared with 79.3% of clinician assessments. Under-triage occurred in 3.6% of LLM outputs and 16.4% of clinician outputs. Unsupported or erroneous LLM content occurred in 1.8% (52/2880) of outputs, and potentially unsafe outputs occurred in 0.4% (12/2880).

Conclusions:

In a controlled, fixed-information simulation, LLMs used as a structured intake layer more consistently preserved DDH-relevant risk signals and mapped them to expert-defined triage actions. This does not establish clinical screening performance, diagnostic accuracy, or autonomous safety. The next step should compare usual intake, a conventional checklist, and supervised LLM-assisted intake in real or near-real pediatric workflows.