JMIR Publications

ABSTRACT

Background:

Primary care encounters are increasingly constrained by limited time and documentation burden, reducing opportunities for meaningful patient–physician communication. Pre-visit planning tools can improve efficiency but are often rigid, burdensome, and inconsistently adopted. Patient-facing applications using large language models (LLMs) offer the potential for more flexible, conversational approaches to eliciting patient information prior to clinical encounters, though concerns remain regarding safety, hallucination, and workflow integration.

Objective:

To evaluate the feasibility of an LLM-based conversational assistant (PCP-Bot) for pre-visit planning in primary care, focusing on the quality, usability, and perceived clinical utility of generated pre-visit summaries.

Methods:

We conducted a prospective feasibility study using simulated primary care scenarios. PCP-Bot, implemented using ChatGPT-4o, engaged users via a voice-based conversational interface and generated structured pre-visit summaries using a schema-constrained output. Ten synthetic cases were enacted by trained non-medical researchers acting as patients, producing 30 complete dialogues and corresponding summaries. Practicing physicians (N=10) independently rated summaries across six domains (usefulness, readability, relevance, coherence, comprehensiveness, and factual accuracy) using 5-point Likert scales. Perceived usefulness (TAM-PU) was assessed among physicians, and perceived ease of use (TAM-PEU) was assessed among participants simulating patient interactions. Quantitative analyses examined interaction characteristics and their associations with summary quality.

Results:

PCP-Bot generated concise conversations (median 28 exchanges, IQR 26.25–31) and summaries (median 148 words, IQR 132.75–162). Clinician ratings were favorable across domains, including usefulness (mean 3.99, SD 0.25), relevance (mean 4.07, SD 0.21), readability (mean 4.07, SD 0.26), coherence (mean 3.94, SD 0.27), and comprehensiveness (mean 3.88, SD 0.22), with a low hallucination rate (mean 0.51, SD 0.25). Simulated patients reported high perceived ease of use (mean TAM-PEU 97.2, SD 2.48), while physicians reported moderate perceived usefulness (mean TAM-PU 61.3, SD 15.9). Longer summaries were associated with higher ratings of usefulness (r=0.39, P=.033) and comprehensiveness (r=0.39, P=.031), whereas longer patient dialogue was negatively associated with perceived relevance (r=−0.39, P=.035).

Conclusions:

In simulated primary care scenarios, an LLM-based conversational assistant produced concise, structured pre-visit summaries that clinicians rated favorably, supporting the feasibility of conversational pre-visit workflows. Summary quality appears sensitive to the balance between detail and conciseness. Real-world evaluation is needed to assess clinical impact, safety, equity, and integration into routine care.