JMIR Publications

ABSTRACT

Large language models (LLMs) are increasingly embedded in clinical and population health workflows, including conversational agents such as health chatbots. As chatbots evolve from rule-based approaches to hybrid and LLM-enabled designs, risks and worries about deployment readiness shift. Unlike rules-based chatbots, LLM outputs can be unpredictable, error-prone, and difficult to validate with traditional evaluation methods. Public health teams integrating customized LLMs into interventions face practical and ethical challenges related to performance variability, uncertainties about model behaviors, and inequitable performance across languages. Although existing frameworks address domains such as safety, ethics, effectiveness, engagement, and implementation, they often assume or imply, rather than operationalize, an explicit benchmark for deployment and implementation decisions. We propose an acceptance-criteria framework (ACF) to determine implementation fit, defined as meeting prespecified minimum performance standards and demonstrating non-problematic behavior under anticipated use. The ACF uses project-relevant and off-topic prompts, structured expert review, and prespecified thresholds to produce a documented decision record that can be iteratively re-run after model revisions. We demonstrate the framework through a case application in a tobacco cessation text messaging intervention, illustrating how the ACF can guide deployment decisions.