Accepted for/Published in: JMIR Formative Research
Date Submitted: Mar 31, 2026
Date Accepted: Jun 15, 2026
AI-Assisted Clinical Data Abstraction from Electronic Health Records: Retrospective Concordance Study
ABSTRACT
Background:
Manual chart abstraction is a critical step in clinical outcomes research but is time-intensive and prone to human error. Advances in artificial intelligence (AI), particularly large language models, offer the potential to automate extraction of structured data from unstructured electronic health records (EHRs) with improved efficiency and consistency.
Objective:
To evaluate the accuracy and efficiency of an AI-assisted approach for extracting patient-reported outcomes from clinical notes compared with traditional human abstraction.
Methods:
We conducted a retrospective study of 26 patients treated with low-dose radiation therapy for osteoarthritis. Human reviewers abstracted Numeric Rating Scale (NRS, 0–10) pain scores and von Pannewitz Score (VPS, 0–4) improvement scores at baseline, end of treatment, and first follow-up. A HIPAA-compliant GPT-based AI system was prompted to extract the same endpoints from clinical notes. Concordance was assessed using exact match rates, intraclass correlation coefficient (ICC) for NRS, and weighted Cohen’s kappa for VPS. Time required for AI versus manual abstraction was recorded.
Results:
The AI demonstrated high concordance with human abstraction, achieving 92% exact matches for NRS (ICC = 0.96) and 94% for VPS (κ = 0.91). All discrepancies were minor, and no spurious values were generated. The AI identified one clinically relevant data point missed during manual review. Average abstraction time per patient decreased from approximately 30 minutes to 2 minutes, representing over 90% time savings. The system also successfully captured trends in analgesic use, including reductions without escalation.
Conclusions:
AI-assisted data abstraction achieved near-human accuracy for extracting patient-reported outcomes from EHR clinical notes while substantially reducing time and labor. This approach provides a scalable and practical solution to improve efficiency and consistency in clinical research workflows and is broadly generalizable to other domains requiring structured data extraction from unstructured medical records.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.