JMIR Publications

ABSTRACT

Background:

Depression is rising among people aged 10–24. Traditional depression screening methods, such as the PHQ-9, are particularly challenging for children. AI has potential to help but the scarcity of annotated datasets highlights the need for zero-shot approaches. In this work, we investigate the feasibility of state-of-the-art Large Language Models (LLMs) for depressive symptom extraction in pediatric settings. This approach aims to complement traditional screening.

Objective:

The key objectives were to: 1) Assess the feasibility of zero-shot LLMs in identifying depressive symptoms in free-text clinical notes for pediatric populations, 2) Benchmark the zero-shot performance of leading LLM models in extracting PHQ-9 symptom-related information, 3) Demonstrate the potential clinical value of LLM-driven evidence extraction to improve mental health screening using an example of an interpretable AI-based depression screening tool.

Methods:

We examined free text of the EHRs of pediatric patients with the diagnosis of depression or related mood disorders (age groups 6-24, 1.8K patients) from the Cincinnati Children's Hospital Medical Center. We noticed drastic inconsistencies in the application and documentation of PHQ-9 screening highlighting the difficulty in obtaining comprehensive diagnostic data in these conditions. We manually annotated notes for 22 patients with 16 depression-related symptom categories. We leveraged the combination of Beck's Depression Inventory (BDI) and the PHQ-9 to develop tailored categories specifically suited for pediatric depression symptoms. We then applied three state-of-the-art Large Language Models (LLMs) (FLAN T5, Llama and Phi) to automate the identification of these symptom categories.

Results:

Our findings show that all LLMs are 60% more efficient than word match with Flan leading in precision (average F1: 0.65, precision: 0.78), excelling in the extraction of more rare symptoms like "sleep problems" (F1 0.92) and "self-loathing" (F1 0.8). Phi strikes a balance between precision (0.44) and recall (0.60). Llama 3 with the highest recall (0.90), overgeneralizes symptoms and is less suitable. Challenges include the complexity of clinical notes and overgeneralization from PHQ-9 scores. We finally demonstrate the utility of symptom annotations provided by Flan as features in an ML algorithm which differentiates depression cases from controls with high precision of 0.78 with major performance boost as compared to a baseline not using these features.

Conclusions:

This study highlights the strengths of LLMs in addressing data scarcity and heterogeneity by extracting depressive symptoms from free text of pediatric EHRs with high precision. The computational efficiency of FLAN-T5 further supports its potential for deployment in resource-limited clinical settings. Our findings are constrained to the pediatric age group, requiring further validation for broader populations and other mental health conditions. Our study demonstrates the potential of LLMs to enhance depression screening, improve diagnostic consistency, and provide an interpretable tool for mental health clinicians.