JMIR Publications

ABSTRACT

Background:

Documentation burden is a major contributor to physician burnout, and AI-assisted documentation tools have emerged as a potential solution. While early evidence suggests these tools reduce documentation time in medical settings, psychiatry presents distinct challenges, as notes capture sensitive information about mental states, suicidal ideation, and psychosocial circumstances. These tools are already entering clinical use, yet the quality of AI-generated documentation has not been thoroughly evaluated in psychiatry. To our knowledge, no prior study has examined psychiatrists' perceptions of the quality of AI-generated psychiatric SOAP (Subjective, Objective, Assessment, and Plan) notes.

Objective:

We aimed to understand in detail how psychiatrists perceive the quality of unedited AI-generated SOAP notes using a controlled design, focusing on six notes generated by three frontier large language models for a single simulated telehealth encounter.

Methods:

We conducted a prospective, mixed-methods exploratory study in which 12 US-based, board-certified or board-eligible psychiatrists evaluated six AI-generated SOAP notes about a single simulated psychiatric telehealth encounter. The notes were generated by three frontier LLMs (OpenAI GPT-5, Anthropic Claude Sonnet 4, and Google Gemini 2.5 Pro) from the session transcript using an identical prompt. Evaluators watched the session recording, then rated each note on the 10-attribute Modified Physician Documentation Quality Instrument, which has been used in other AI-generated clinical note quality studies. We also collected written rationales for each attribute rating and an overall comment for each note. Finally, evaluators completed a post-evaluation survey. Rationales and survey responses were analyzed using content and thematic analysis. Note ratings and coded qualitative data were analyzed together to understand psychiatrists’ perceptions. We also assessed agreement among evaluators using intraclass correlation coefficients and Kendall’s coefficient of concordance.

Results:

Across 72 evaluations (12 evaluators x 6 notes), notes received a mean total score of 42 of 50 (median 42, IQR 39-46). Despite favorable ratings, written rationales revealed concerns of verbosity, factual errors and hallucinations (including two notes with incorrect session durations), and issues with the mental status exam sections. We found that highly rated notes contained errors that may be consequential if not caught during review, and not all evaluators caught notable errors in the notes in this study. While the notes were not perfect, evaluators simultaneously praised their usefulness and thoroughness. Interestingly, inter-evaluator agreement on total scores was poor (ICC=0.04), and agreement on the individual quality attributes was similarly low, ranging from -0.04 to 0.17. Analysis of variance components revealed that variance was attributed more to differences among evaluators than differences among the notes, which may indicate differences in opinion among psychiatrists as to what constitutes quality documentation. Comments revealed similar differences across evaluators.

Conclusions:

The AI-generated notes in this study were perceived favorably by psychiatrists, who highlighted their usefulness and acknowledged their imperfections. Psychiatrists showed little agreement with one another about which notes were better and qualitative data revealed they prioritized different aspects of the notes in their comments. While our specific findings are not generalizable, they do underscore the perceived usefulness of AI-generated notes in psychiatry, as well as the necessity for thorough review before signing them.