JMIR Publications

ABSTRACT

Background:

Systematic collection of social determinants of health (SDoH) data remains inconsistent across healthcare settings, despite its critical impact on patient outcomes. Large language model (LLM)-powered chatbots offer promise for scalable SDoH data collection, but rigorous, feasible evaluation methods for patient-facing applications are lacking.

Objective:

To describe an efficient, iterative, multidisciplinary approach for developing and evaluating a patient-facing SDoH chatbot using synthetic data and case simulation, with the goal of optimizing both chatbot performance and the evaluation rubric prior to clinical deployment.

Methods:

A 10-criterion evaluation rubric was adapted from established healthcare artificial intelligence (AI) frameworks and applied to 27 synthetic clinical scenarios representing diverse SDoH profiles. Scenarios were role-played by a licensed clinical social worker, and chatbot-patient interactions were independently rated by three multidisciplinary experts (social worker, nurse practitioner, physician). Quantitative analysis used descriptive statistics and percent agreement to characterize chatbot performance and rater consensus, with percent agreement selected due to high prevalence of ceiling effects in several domains. Qualitative analysis synthesized rater feedback to guide iterative refinement of both chatbot prompts and rubric domains.

Results:

The chatbot demonstrated robust performance in domains such as accurate interpretation (mean = 0.98, SD = 0.09), communication quality and cultural sensitivity (mean = 0.99, SD = 0.06), and adaptive questioning (mean = 0.99, SD = 0.06), with near-perfect rater agreement. Lower scores and greater variability were observed in completeness of data collection, systematic domain exploration, and safety, prompting targeted adaptations. Qualitative feedback highlighted the importance of distinguishing screening from clinical interviewing capabilities and informed the refinement of the rubric, including clarifying the definition of safety to focus on recognition of physical and mental health emergencies.

Conclusions:

This study provides a practical, replicable blueprint for pre-deployment evaluation of patient-facing SDoH chatbots, balancing rigor with feasibility. The iterative, multidisciplinary approach enabled rapid identification and remediation of performance gaps, supporting responsible integration of AI into SDoH data collection. Explicit performance thresholds and rubric refinement are essential for protecting patient trust and safety, particularly in vulnerable populations. Future work will validate findings with real patient interactions and expand stakeholder involvement. Clinical Trial: N/A