JMIR Publications

ABSTRACT

Background:

Colorectal cancer screening relies on structured risk assessment and guideline-concordant communication, which remain challenging to implement consistently in real-world practice. Digital tools based on large language models may support such workflows, but their feasibility and safety in structured screening contexts have not been well evaluated.

Objective:

To develop and evaluate a guideline-concordant chatbot framework for structured colorectal cancer screening communication.

Methods:

A multi-stage feasibility study was conducted. In Phase I, baseline performance of contemporary large language models was assessed using 14 standardized colorectal cancer screening questions and validated expert-rated instruments. In Phase II, structured prompt versions were iteratively optimized based on screening guidelines and expert feedback and tested using simulated user scenarios. In Phase III, the optimized chatbot was evaluated in 50 adult screening-eligible clinical participants to assess feasibility, safety, and guideline concordance.

Results:

Across initial assessments, language models demonstrated stable baseline performance. Iterative prompt optimization was associated with improvements in dialogue-level performance, particularly in risk information completeness, communication clarity, and safety (all p < 0.001). In real-world evaluation, the optimized chatbot achieved high completeness of risk information collection and provided guideline-concordant screening recommendations for nearly all participants, with no inappropriate or unsafe outputs observed.

Conclusions:

This study demonstrates the feasibility and safety of a structured, guideline-concordant chatbot framework for colorectal cancer screening communication. The proposed approach offers a reproducible pathway for integrating large language models into preventive screening workflows and supports future evaluations of clinical impact and implementation at scale.