Previously submitted to: JMIR Medical Informatics (no longer under consideration since Aug 06, 2026)
Date Submitted: Jul 14, 2026
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Generative Artificial Intelligence Assisted Manual Chart Review for Substance Use Disorder Phenotyping: A Crossover Study
ABSTRACT
Background:
Artificial intelligence (AI) and natural language processing tools have been proposed as approaches to improve the efficiency of electronic health record–based phenotyping. However, these tools have not been routinely evaluated for complex case definitions pertaining to substance use disorders (SUDs), which often require detailed manual chart review and clinical interpretation.
Objective:
This study aimed to determine how an AI-assisted chart review tool compares with manual review alone for SUD phenotyping.
Methods:
We conducted a two-period, two-treatment crossover study using 20 randomly selected patients from a parent study cohort in the Vanderbilt University Synthetic Derivative, a near–population-level deidentified research database. Seven SUD categories were assessed: tobacco, alcohol, opioids, cannabis, stimulants, sedatives, and other substances. Treatment A consisted of human manual chart review alone, and Treatment B consisted of human manual chart review with AI assistance. Two reviewers acted as their own independent controls by reviewing a fixed set of 10 patients with one randomly assigned treatment and the remaining 10 patients with the other treatment. Outputs were compared with a reference standard derived from adjudicated human review. Primary outcomes included inter-rater agreement, Cohen’s κ, precision, recall, and F1 score. The secondary outcome was efficiency, measured as average time saved per patient record.
Results:
Of the 20 patients, 13 (65.0%) had evidence of problematic substance use according to the reference standard. Inter-rater agreement was high, with 91.4% agreement and a Cohen’s κ of 0.86. Compared to the reference standard, human review with AI assistance had the highest F1 score (0.89), followed by human review alone (0.83) and the AI tool alone (0.80). Reviewers saved an average of 2.7 minutes per patient record when using the AI tool compared with manual review alone (P=.03).
Conclusions:
AI-assisted manual chart review improved both performance and efficiency for SUD phenotyping in this crossover study. These findings suggest that scalable SUD phenotyping may benefit from careful integration of human expertise within semiautomated AI-assisted review workflows. Clinical Trial: N/A
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.