JMIR Publications

ABSTRACT

Background:

Suicide is the second-leading cause of U.S. childhood mortality after age nine. Accurate measurement of pediatric emergency service use for self-injurious thoughts and behaviors (SITB) remains challenging, as diagnostic codes undercount children. This measurement gap impedes public health and prevention efforts. Current research has not established which combination of electronic health record (EHR) data elements achieves both high detection accuracy and consistent performance across youth populations.

Objective:

To 1) compare detection accuracy of EHR-based methods for identifying SITB-related pediatric emergency department (ED) visits: basic structured data (ICD-10-CM codes, chief complaint), comprehensive structured data, clinical note text with natural language processing, and hybrid approaches combining structured data with notes; and 2) for each method, measure variability in detection by youth demographics and underlying mental health diagnosis.

Methods:

Multiple human experts reviewed clinical records of 3,828 pediatric mental health emergency visits (28,861 clinical notes) to a large health system with two EDs (June 2022-October 2024). Reviewers used the Columbia Classification Algorithm for Suicide Assessment (C-CASA) to label presence of SITB at the visit. Random forest classifiers were developed using three data modalities: 1) structured data (low-dimensional [ICD/CC], medium-dimensional [adding c-SSRS screening or mental health diagnoses], and high-dimensional [all structured data/aCS]); 2) text data (NLP-general, NLP-medical, and LLaMA-derived scores); and 3) hybrid data (combining aCS with each text approach). Model performance was evaluated using area under the receiver operating characteristic curve (AUROC).

Results:

Of 3,828 visits, 1,760 (46.0%) were SITB-related. Detection performance improved with dimensionality: low- (AUROC=0.865), medium- (AUROC=0.934-0.935), and high-dimensional (AUROC=0.965). Low-dimensional structured (ICD/CC) showed high variability in detection, with lower accuracy among preadolescents (AUROC=0.821 versus 0.880 for adolescents), males (AUROC=0.817 versus 0.902 for females), and patients with neurodevelopmental (AUROC=0.568-0.809), psychotic (AUROC=0.718), and disruptive disorders (AUROC=0.703). Hybrid modality (aCS+LLaMA) achieved optimal performance (AUROC=0.977), with AUROC ≥0.90 for all 20 demographic and 12/15 diagnostic subgroups.

Conclusions:

Relative to diagnostic codes and chief complaint alone, hybrid structured-text detection methods greatly improved accuracy, mitigated unwanted detection variability, and may offer a more robust scaffold for information retrieval of pediatric suicide and self-harm-related emergencies. Clinical Trial: N/A