JMIR Publications

ABSTRACT

Background:

The accurate detection of clinical phenotypes from electronic health records (EHRs) is pivotal for advancing large-scale genetic and longitudinal studies in psychiatry. Free-text clinical notes are an essential source of symptom-level information, particularly in psychiatry; however, the automated extraction of symptoms from clinical text remains challenging.

Objective:

We aimed to evaluate the ability of open-source generative large language models (LLMs) to detect psychiatric phenotypes from clinical text in Spanish, and to develop a new fine-tuned, open-sourced LLM that can be shared without risk of exposing patient information.

Methods:

We tested 11 open-source generative large language models (LLMs) for their ability to detect 109 psychiatric phenotypes from clinical text, using annotated EHR notes from a psychiatric hospital in Colombia. The LLMs were evaluated both “out-of-the-box” and after fine-tuning, and compared against a traditional natural language processing (tNLP) method developed from the same data. To develop a fine-tuned LLM that can be shared without risk of exposing patient information, we created a novel synthetic dataset, leveraging real phenotype annotations while generating fully synthetic clinical text, and fine-tuned a top-performing LLM on this data. Fine-tuned model generalizability was assessed through cross-site evaluation using data from a second independent hospital in Colombia.

Results:

We show that while base LLM performance was poor to moderate (0.2-0.6 macro-F1 for zero-shot; 0.2-0.74 macro-F1 for few shot), it improved significantly after fine-tuning (0.75-0.86 macro-F1), with several fine-tuned LLMs outperforming the tNLP method. In total, 100 phenotypes could be reliably detected (F1>0.8) using either a fine-tuned LLM or tNLP. Using the synthetic dataset, we developed Mistral-small-psych, a new LLM capable of detecting psychiatric phenotypes from Spanish clinical text with performance comparable to models trained on real EHR data (macro-F1=0.79) while remaining fully distributable. In cross-site evaluations, most LLMs generalized well, with small out-of-domain performance losses (macro-F1 decrease of 0.02–0.16).

Conclusions:

This work provides the first large-scale, cross-institutional evaluation of state-of-the-art LLMs for psychiatric phenotyping in Spanish EHRs, introduces a privacy-preserving pathway for building shareable clinical NLP models, and presents a new open model that supports globally inclusive precision psychiatry.