JMIR Publications

ABSTRACT

Background:

Mental health clinical notes contain decision-critical information often absent from structured electronic health record (EHR) fields. Large language models (LLMs) can extract clinically relevant signals from narrative text; however, variability in output format, limited reproducibility, and inconsistent evaluation remain barriers to clinical deployment. Despite rapid advances in LLM-based information extraction, clear and reproducible guidance for interdisciplinary clinical teams is limited.

Objective:

This tutorial presents a structured workflow for zero-shot information extraction from mental health clinical notes using locally deployed open-source LLMs. It aims to reduce barriers for clinicians and researchers with limited familiarity with natural language processing (NLP) or LLM pipelines. Each stage includes key decision points and examples. The workflow is illustrated on two tasks: (1) detection of self-injurious thoughts and behaviors (SITB) in pediatric emergency department (ED) notes and (2) antipsychotic medication nonadherence detection in outpatient notes, using schema-constrained outputs and standardized evaluation.

Methods:

We describe a five-stage zero-shot LLM pipeline: (1) infrastructure setup with local deployment via Ollama to prevent Protected Health Information (PHI) transmission; (2) task definition specifying the clinical construct, output format, and evaluation; (3) dataset preparation using de-identified or synthetic notes; (4) iterative prompt development using a held-out set with binary and Likert-scale outputs constrained via JSON schemas; and (5) output parsing, normalization, and validation. We generated 200 synthetic notes per task using separate LLMs for generation and evaluation. A hold-out set of 40 notes per task was used for prompt iteration and excluded from final metrics. Evaluation used Llama 3.2 and Llama 3.3 with deterministic decoding (temperature=0). Performance was assessed using accuracy, precision, recall, and F1 score; Likert thresholds were optimized using Youden’s index with bootstrapped confidence intervals (CIs).

Results:

Llama 3.3 outperformed Llama 3.2 across both tasks. For pediatric SITB detection (n=200; 50% positive), Llama 3.3 achieved 97.5% accuracy with balanced precision and recall; misclassifications illustrate how error analysis informs prompt refinement. For medication nonadherence detection (n=200; 30% nonadherent), Llama 3.3 achieved 100% accuracy, precision, and recall using Likert-based thresholding; this reflects the simplicity of the synthetic dataset rather than expected real-world performance. JSON-based constraints reduced parsing errors and improved robustness. Likert prompting preserved graded uncertainty, enabling threshold tuning based on clinical risk tolerance.

Conclusions:

A schema-driven, zero-shot LLM workflow can support reproducible extraction of clinically relevant information from narrative notes. Local deployment enables processing without transmitting PHI to external servers. This tutorial provides a transferable methodology for institutional adaptation and validation prior to clinical use. All prompts, code, and datasets are publicly available via Zenodo. Clinical Trial: N/A