JMIR Publications

ABSTRACT

Background:

Self-harm, defined as intentional self-injury or self-poisoning irrespective of motivation, is the strongest risk factor for suicide and an important outcome for mental health care. Although highly prevalent in clinical populations, it is often imprecisely captured in routinely collected clinical data, where it is typically recorded and stored in unstructured, free-text format. Contemporary language models, such as GPT (OpenAI), Gemini (Google) and Claude (Anthropic) can analyse free-text clinical notes, but such cloud-based commercial and closed-source models may violate data governance of processing sensitive patient data.

Objective:

We evaluated whether a privacy-preserving language model running entirely within an institution’s secure computing infrastructure (here, the UK National Health Service) could accurately identify the presence and timing of self-harm using electronic health records (EHRs) from secondary mental healthcare.

Methods:

A random sample of 1,352 clinical notes from the EHRs of people with a confirmed diagnosis of a psychiatric disorder, according to the International Classification of Diseases (ICD-10), was selected from Oxford Health NHS Foundation Trust. Each clinical note was annotated for (i) presence or absence of self-harm and (ii) its recency (≤90 days vs. >90 days vs. unknown), constituting the gold-standard data set for model development and validation. Privacy-preserving locally served 27-billion-parameter Gemma 3 language model (‘Gemma3-27b’) was used as a core language model to identify self-harm along with its timing to generate a structured output per clinical record. The performance of Gemma3-27b model was compared against a strong baseline multi-label text classification model based on RoBERTa (Robustly Optimized BERT Pretraining Approach, a transformer-based language model) architecture. Model performance was evaluated using precision, recall, and the F1-score (harmonic mean of precision and recall), with 95% confidence intervals estimated from 1,000 bootstrap samples with replacement.

Results:

The Gemma3-27b model outperformed the RoBERTa classifier across all categories. Gemma3-27b achieved Precision = 0.92, Recall = 0.92 (sensitivity), and F1-score of 0.92 for notes containing self-harm, and Precision = 0.97, Recall = 0.97 (specificity), and F1-score of 0.97 for notes without self-harm. The performance of Gemma3-27b for recent self-harm was Precision = 0.84, Recall = 0.75, and F1-score of 0.79. The global weighted F1-score of Gemma3-27b across all categories was 0.88, compared to 0.85 for RoBERTa.

Conclusions:

The prompt-engineered local language model outperformed the multi-label text classification RoBERTa model on the task of ascertaining self-harm events and their timing, while requiring significantly fewer high-fidelity annotated data. This approach offers a practical solution for improving self-harm case ascertainment whilst adhering to strict data governance rules to potentially transform clinical monitoring and response to this critical public health challenge. Clinical Trial: None