JMIR Publications

ABSTRACT

Background:

The COVID-19 pandemic has exacerbated depression, which is recognized as a significant social and medical concern. With the increasing interest in generative AI as an interactive consultant, there is a need to assess its applicability in medical discussions and consultations, particularly in the domain of depression.

Objective:

This study aims to evaluate the capability of large language models (LLMs) in generating responses to depression-related questions and compare the similarity between the generated and original answers.

Methods:

Depression-related questions and corresponding answers were collected from PubMedQA and QuoraQA datasets. Four LLMs (BioGPT, PMC-Llama, GPT-3.5, and Llama2) were utilized to generate responses to these questions. The quantity and quality of the generated responses were assessed, and the similarity between the quality answers and the correct answers was measured using cosine similarity.

Results:

The latest general LLMs, GPT-3.5 and Llama2, outperformed the biomedical domain LLMs (BioGPT and PMC-Llama) in generating answers to medical questions sourced from PubMedQA. GPT-3.5 and Llama2 generated answers with higher similarity to the original answers, with scores of 0.632 and 0.590, respectively. The textual validity of the generated answers was also deemed satisfactory.

Conclusions:

The rapid development of LLMs in recent years suggests that version upgrades of general-purpose models offer greater benefits in enhancing the capacity to generate "knowledge text" in the biomedical domain compared to fine-tuning for the biomedical field specifically. The findings highlight the potential for future advancements in prompt engineering and interactive process modeling to further enhance the capability of general LLMs in generating responses to biomedical questions.