JMIR Publications

ABSTRACT

Background:

Depressive disorders have substantial global implications, leading to various social consequences, including decreased occupational productivity and a high disability burden. Early detection and intervention for clinically significant depression have gained attention; however, the existing depression screening tools, such as the Center for Epidemiologic Studies Depression Scale, have limitations in objectivity and accuracy. Therefore, researchers are identifying objective indicators of depression, including image analysis, blood biomarkers, and ecological momentary assessments (EMAs). Among EMAs, user-generated text data, particularly from diary writing, has emerged as a clinically significant and analyzable source for detecting or diagnosing depression, leveraging advancements in large language models such as Chat Generative Pre-trained Transformer (ChatGPT).

Objective:

To detect depression based on user-generated diary words through an emotional writing application utilizing a large language model (LLM). We aimed to validate the value of the semi-structured diary text data as an EMA data source.

Methods:

Individuals who voluntarily participated in the experiment were assessed for depression using the Patient Health Questionnaire and suicide risk using Beck's Suicide Ideation Scale, along with daily diary entries before and after diary writing. The performance of leading LLMs, such as ChatGPT 3.5 and 4, was assessed with and without GPT 3.5 fine-tuning on the training dataset. The model performance comparison involved the use of chain-of-thought and zero-shot prompting to analyze the text structure and content.

Results:

428 diaries from 91 participants were used; GPT 3.5 fine-tuning demonstrated superior performance in depression detection, achieving an accuracy of 0.902 and a specificity of 0.955. However, the balanced accuracy was the highest (0.844) in GPT 3.5 without fine-tuning and prompt techniques; it displayed a sensitivity of 0.929.

Conclusions:

Both GPT 3.5 and GPT 4.0 demonstrated relatively reasonable performance in recognizing the risk of depression based on diary entries. Our findings highlight the potential clinical usefulness of user-generated text data for detecting depression. In addition to measurable indicators, such as step count and physical activity, future research should increasingly emphasize qualitative digital expressions. Clinical Trial: The study methods adhered to the ethical principles outlined in the tenets of the Declaration of Helsinki and were approved by the Institutional Review Board of Korea University Anam Hospital (2023AN0379).