JMIR Publications

ABSTRACT

Background:

Spoken virtual patients enable medical learners to practise clinical history-taking through spoken questions and patient replies across a full consultation. A major challenge is turn-level fluency accompanied by gradual drift from the intended case, including changing disclosed symptoms, adding unsupported clinical details, or losing persona consistency. Retrieval-augmented generation (RAG), slot-based dialogue state tracking, and transcript replay mainly provide inference-time context and do not train the model to preserve the patient's clinical and interaction trajectory.

Objective:

This study aimed to develop and evaluate SpeechTrace, a trajectory-supervised speech-to-speech virtual patient designed to preserve clinical and interaction trajectories during long-horizon history-taking.

Methods:

SpeechTrace learns a fixed-size trajectory state representing disclosed clinical information and interaction behaviour. During training, this state is supervised with reference signals. At inference, the state is updated recurrently from learner speech and mapped to a decoder prefix for spoken generation without an automatic speech recognition (ASR) transcript.

Results:

On held-out interactive MedTOD dialogues (740 scored turns), full-parameter SpeechTrace improved reference alignment over the Base spoken patient by 4.3% at 0.5B and 3.4% at 3B, reduced unsupported clinical claims, and improved persona and coherence scores; a small LoRA adapter (3B backbone frozen) did not. On 1393 doctor turns from 15 cases (RTX 5090, FlashAttention v4, R=3/W=10), the 3B SpeechTrace model achieved a median time-to-first-audio (TTFA) of 455 ms, versus 1451 ms for matched ASR-LLM-TTS (approximately 3.2× higher) and 567.8 ms reported for LLaMA-Omni2-3B; median full-reply latencies were 1623 ms versus 2806 ms. Ablations indicated that most gains arose from training-time trajectory supervision when short dialogue history was available, whereas recurrent state carry was essential without prior-turn history. Adding RAG or an oracle dialogue-state slot table at test time helped more when combined with SpeechTrace than with Base alone, with scale-dependent gains.

Conclusions:

Turn-level fluency and inference-time memory alone are insufficient for long-horizon spoken virtual patients. Trajectory-supervised training improves cross-turn consistency, while end-to-end speech-to-speech generation remains substantially faster than cascaded ASR-LLM-TTS. Findings are based on GPT-based automatic judgments; future OSCE-style human evaluations should assess educational realism and clinical validity.