Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently submitted to: Journal of Medical Internet Research

Date Submitted: Nov 20, 2026
Open Peer Review Period: Nov 21, 2026 - Jan 16, 2027
(currently open for review)

Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.

Theory-Grounded Evaluation of Multi-Turn Large Language Model Communication for Cancer Screening: Framework Development and Evaluation Study

  • Agnik Saha; 
  • Victoria Churchill; 
  • Alexander Eaves; 
  • Ugur Kursuncu; 
  • Muhammed Y. Idris

ABSTRACT

Background:

Counseling a person about cancer screening requires more than factually accurate information. It must respond to the person's readiness to act, beliefs, barriers, emotions, and sociocultural context while conveying guideline-concordant recommendations. Large language model (LLM)–based health communication is often evaluated with general measures of fluency, coherence, or isolated factual accuracy, which may not show whether a multi-turn conversation is clinically accurate, complete, or patient centered.

Objective:

This study aimed to develop a theory-grounded framework for structuring and evaluating multi-turn LLM-based breast and cervical cancer screening communication and to apply it to identify where conversational quality diverges from clinically relevant and patient-centered quality.

Methods:

Constructs from the Transtheoretical Model and the Health Belief Model, intrapersonal and interpersonal (sociocultural) determinants of screening, patient-centered communication functions, and age-specific screening guidance were operationalized in a structured knowledge representation with patient and provider branches. Role-specific GPT-4o Patient and Provider Agents instantiated this representation as a controlled simulation testbed, with stage-aware provider communication, dialogue state tracking, and retrieval of guideline text, and generated 2000 synthetic patient-provider dialogues. The synthetic conversations were not assumed or validated to represent real patient-provider conversations. Dialogues were evaluated with Fine-grained Evaluation of Dialog (FED), rubric-based G-Eval, and human review of 100 selected dialogues by two annotators on six accuracy, completeness, and communication criteria, each scored from −1 to 1.

Results:

FED scores were highest for coherence (0.98), error recovery (0.97), and consistency (0.94) and lowest for likeability (0.45) and informativeness (0.47). G-Eval scores ranged from 2.34 to 2.50 on a 1-3 scale. A total of 83 dialogues had complete paired human ratings, with a pooled composite score of 0.59 (95% CI 0.56-0.63). Among the human-rated criteria, patient-centered concern had the highest pooled mean (0.87) and cancer screening accuracy the lowest (0.28), followed by response completeness (0.46). Overall interannotator agreement was fair (linear-weighted Cohen κ=0.29); criterion-level agreement was highest for cancer screening accuracy (κ=0.52) and near zero for emotional acknowledgment (κ=0.08) and patient-centered concern (κ=−0.05).

Conclusions:

Behavioral and communication theory can be operationalized into an interpretable structure for examining multi-turn LLM cancer screening communication. In this evaluation, high automated scores for conversational continuity coexisted with low human ratings of cancer screening accuracy and response completeness. Conversational fluency and coherence should therefore not be interpreted as evidence of clinically accurate or patient-centered health communication. The findings come from one model configuration, synthetic dialogues, and two annotators, without a baseline or ablation; validation against real conversations, by clinicians and community members, and against patient outcomes is needed before any patient-facing use.


 Citation

Please cite as:

Saha A, Churchill V, Eaves A, Kursuncu U, Idris MY

Theory-Grounded Evaluation of Multi-Turn Large Language Model Communication for Cancer Screening: Framework Development and Evaluation Study

JMIR Preprints. 20/11/2026:117464

DOI: 10.2196/preprints.117464

URL: https://preprints.jmir.org/preprint/117464

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.