JMIR Publications

ABSTRACT

Background:

Understanding the clinical course of psychiatric disorders is vital for informed decision-making. Extracting details such as onset time, episode count, and hospitalization history from unstructured psychiatric notes remains challenging.

Objective:

This study evaluates the performance of large language models (LLMs)—LLaMA, MentaLLaMA, OpenBioLLM, and Mistral—in extracting temporal and event-based clinical information from psychiatric discharge summaries.

Methods:

We used 500 annotated discharge summaries from the NTUH-iMD, covering psychiatric diagnoses (ICD-9-CM: 290–319, ICD-10-CM: F00–F99). Key temporal and clinical course features were annotated by a psychiatrist and an NLP researcher. A two-stage extraction process was implemented: first, sentence-level models identified clinical events and temporal cues; then, a chart-level model predicted four clinical course features: onset time, episode count, number of hospitalizations, and most recent hospitalization time. Performance was evaluated using F1-scores.

Results:

Among 12,947 analyzed sentences, 7,177 included clinical events and 4,842 contained temporal information. Mistral achieved the best performance in event extraction (episodes: 0.968; hospitalizations: 0.933; remission/response: 0.901) and temporal information (age: 0.968; time expressions: 0.967; duration: 0.901). In chart-level extraction, F1-scores were highest for Mistral in onset time (0.714), episode count (0.624), number of hospitalizations (0.676), and last hospitalization time (0.842).

Conclusions:

Fine-tuned LLMs, especially Mistral, can accurately extract structured clinical course information from psychiatric notes, offering a scalable alternative to manual review. Future work should address vague temporal expressions, expand feature sets, and validate generalizability across diverse settings. Clinical Trial: None