JMIR Publications

ABSTRACT

Background:

Effective history-taking helps clinicians identify key symptoms and form accurate hypotheses. Generative artificial intelligence (GenAI)-based virtual patients (VPs) are increasingly used to simulate and practice history-taking. However, there is currently no straightforward approach to effectively identify students' clinical reasoning activities during these interactions, which limits the ability to provide instructional feedback.

Objective:

This study aims to develop and validate a coding scheme to identify medical students' history-taking behaviors during interactions with GenAI-based VPs.

Methods:

Second-year medical students (N=210) participated in five history-taking cases with GenAI-based VPs, yielding 1030 dialogs. Researchers applied the systematic text condensation (STC) method to these dialogs data from cases 1 to 4 to inductively develop a coding scheme and validate coding consistency. Subsequently, the dialogs data from case 5 were used to assess the correlation between the students' history-taking behaviors and their academic performance, including diagnostic accuracy, history-taking checklist scores, clinical knowledge test scores, and post-encounter form scores.

Results:

A coding scheme comprising 12 behaviors across 3 dimensions—Clinical Reasoning Behaviors, Information-Gathering Behaviors, and Social-Interactive Behaviors—was developed with high inter-rater reliability (κ ≥ 0.85). The correlation analysis revealed that key Clinical Reasoning Behaviors, such as Summarizing & Integrating and Logical Organization, showed significant positive correlations with multiple performance metrics, underscoring their importance in fostering clinical competence. In contrast, Information-Gathering Behaviors such as Specifying Symptoms and Routine Question were associated with clinical knowledge and history-taking thoroughness, yet were less predictive of diagnostic accuracy.

Conclusions:

This study developed a reliable, theory-informed coding scheme that can identify students' questioning behaviors during history-taking with GenAI-based VPs. The scheme effectively captures higher-order cognitive strategies and provides valuable insights into the development of clinical reasoning in medical students. This approach offers a scalable and efficient way to integrate real-time feedback into future medical education, fostering personalized learning and advancing competency-based assessments in clinical training.