Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Evaluating Large Language Models for Evidence-Linked Quality-of-Care Measurement in Young Children with Attention-Deficit/Hyperactivity Disorder: Retrospective Cohort Study
ABSTRACT
Background:
Learning health systems lack efficient and scalable methods to monitor guideline-concordant care for attention-deficit/hyperactivity disorder (ADHD) and other child mental health conditions as a critical step toward improving health outcomes. Large language models (LLMs) may enable scalable assessment of mental health quality of care by accurately classifying clinical documentation from electronic health records and providing explainable outputs with supporting documentation evidence.
Objective:
To evaluate the performance of LLMs in (1) identifying guideline-concordant first-line parent training in behavior management (PTBM) recommendations documented by pediatricians for young children with ADHD, and (2) providing supporting evidence-linked note excerpts and rationales that allow verification against the source text.
Methods:
We conducted a retrospective cohort study using electronic health records from a community-based pediatric healthcare network in California. The cohort included children aged 4-6 years with ≥2 primary care visits (2020–2024) and ICD-10 diagnoses of ADHD or ADHD symptoms (n=542). Three LLMs (Claude-3.5, GPT-4o, and LLaMA-3.3-70B) were prompted to classify whether the first ADHD-related visit note documented a PTBM recommendation and to return supporting text excerpts with a rationale. Two trained annotators manually reviewed a stratified sample of 122 notes, including all notes with model disagreement, to create a chart-review reference standard. We measured sensitivity, positive predictive value (PPV), and F1-score. Model rationales and extracted evidence were scored using a prespecified qualitative evaluation framework (QUEST).
Results:
All three LLMs performed well, compared with manual chart review. Claude-3.5 demonstrated the most balanced performance (sensitivity=0.89, PPV=0.95, F1-score=0.92) and received the highest scores for evidence-linked rationales. LLaMA-3.3-70B achieved sensitivity=0.91, PPV=0.89, and F1-score=0.90, with the second-highest evidence-linked rationale scores. GPT-4o showed the highest PPV (0.97) but lower sensitivity (0.82) and the lowest evidence-linked rationale scores. Using Claude-3.5 on the full cohort, 26.4% (143/542) of patients had documentation of a PTBM recommendation at the first ADHD-related visit.
Conclusions:
LLMs can support scalable and transparent measurement of guideline-concordant ADHD care in young children by accurately identifying documented PTBM recommendations and returning evidence-linked outputs that can be verified against clinical notes. This approach may enable continuous quality measurement and improvement efforts in pediatric mental health care as part of a learning health system.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.