JMIR Publications

ABSTRACT

Background:

With the increasing burden of chronic diseases, self-management education (SME) is crucial. Traditional SME based on face-to-face oral delivery by clinicians is resource-intensive, and general digital tools, such as web-based platforms, often provide limited interactivity for patient learning. While chatbots based on large language models (LLMs) show promise in usability and interactivity, their real-world effectiveness still lacks empirical evidence.

Objective:

This study aimed to explore the feasibility and preliminary effectiveness of an LLM-based chatbot specifically designed for osteoporosis SME.

Methods:

A formative randomized controlled trial was conducted in a tertiary hospital from February 2024 to March 2025. Adults aged ≥18 years with osteoporosis were recruited and randomly assigned (1:1) to either the intervention (OPBot) group or a control group receiving traditional health education. The chatbot provided interactive educational content and Q&A support, while the control group received face-to-face education and written materials. Osteoporosis knowledge was assessed using the Osteoporosis Knowledge Assessment Tool at baseline and discharge. Nurses’ time spent on health education was self-recorded during each intervention session and aggregated across sessions. Adherence to disease management was assessed at 1, 3, and 6 months post-discharge via telephone using structured Likert-scale questionnaires. The reliability of OPBot responses was evaluated by two clinicians using a 5-point Likert scale, with inter-rater agreement calculated using Cohen’s kappa. Group comparisons were conducted using t-tests, Mann–Whitney U tests, and chi-square tests, and adherence outcomes were analyzed using mixed-effects models.

Results:

A total of 100 participants were randomized; 12 were excluded due to loss to follow-up, refusal of the second knowledge assessment, or death, leaving 88 participants for analysis (45 in the OPBot group and 43 in the control group). The OPBot group showed significantly higher post-intervention knowledge scores than the control group (median [IQR]: 80.0 [70.0–89.0] vs 75.0 [65.5–80.0]; P = .01). Nurses in the OPBot group spent less time on SME than those in the control group (median [IQR]: 5.0 [2.0–17.0] vs 23.0 [20.0–25.0] minutes; P < .001). For adherence outcomes, a significant group × time interaction was observed for calcium supplement intake (OR 1.49, 95% CI 1.08–2.06; P = .02), indicating differing adherence trajectories over time. The OPBot group also showed higher odds of consuming calcium-rich foods across time points (OR 2.87, 95% CI 1.04–7.89; nominal P = .04), although this association did not remain significant after Holm correction for multiple comparisons. No significant effects were observed for sun exposure, exercise, or total adherence scores. In the Q&A module, most OPBot responses were rated as highly reliable (89.4%), with almost perfect inter-rater agreement (Cohen’s κ = 0.83).

Conclusions:

LLM-based chatbots specifically designed for osteoporosis SME may be a feasible approach to improving patient knowledge, supporting adherence behaviors, and reducing healthcare workload. However, further large-scale studies are needed to confirm these findings.