JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) may support clinical decision-making, but evidence for generating complete psychiatric medication regimens from real-world cases remains limited. Evaluation is inherently difficult because multiple regimens may be clinically reasonable for the same patient, leaving no single definitive standard against which generated plans can be judged.

Objective:

We assessed whether supervised fine-tuning on real-world inpatient treatment plans improved the structural concordance and clinician-rated acceptability of LLM-generated psychiatric medication regimens.

Methods:

In this retrospective model-development and validation study, Qwen3-32B was fine-tuned using low-rank adaptation on 23,379 paired case summaries and treatment plans from psychiatric inpatient records. In 172 validation cases, outputs from the base model (Baseline) and fine-tuned model (Finetuned) were compared with electronic health record (EHR)-derived regimens (Reference) using multilevel medication-label concordance, case-level improvement patterns, co-medication structure, and diagnostic subgroup analyses. Three psychiatrists also conducted source-blinded adjudication of Reference, Finetuned, and Baseline regimens in 57 selected cases meeting clinical adjudication criteria.

Results:

At the individual-drug level, mean case-level F1 was 0.587 for Finetuned and 0.410 for Baseline (difference 0.177, 95% confidence interval [CI] 0.129-0.226; Holm-adjusted P<.001). The mean off-diagonal absolute co-occurrence difference from Reference was 0.058 for Finetuned and 0.097 for Baseline (difference -0.039, 95% CI -0.050 to -0.022; P<.001), indicating closer medication-combination structure after fine-tuning. Across 171 case-rater judgments, acceptability was 78.4% (134/171) for Finetuned, 79.5% (136/171) for Reference, and 46.2% (79/171) for Baseline. Finetuned exceeded Baseline by 32.2 percentage points (95% CI 20.5-43.3; P<.001) and did not differ from Reference in paired testing (difference -1.2 percentage points, 95% CI -11.1 to 8.8; P=.90). In forced-choice review, Reference was preferred in 52.0% (89/171) of judgments, Finetuned in 35.1% (60/171), and Baseline in 12.9% (22/171).

Conclusions:

Fine-tuning on real-world inpatient treatment plans improved both prescription-structure concordance and blinded clinician-rated acceptability. In the selected adjudication subset, Finetuned regimens had an acceptability rate numerically close to recorded regimens, although recorded regimens remained more frequently preferred. Combining multilevel structural analysis with blinded clinical adjudication offers a practical evaluation strategy for generative prescribing tasks in which no single regimen can serve as a definitive standard.