JMIR Publications

ABSTRACT

Background:

Standardized patients (SPs) have been a popular assessment method in clinical teaching for decades, even though they are resource-intensive. Nowadays, simulated virtual patients (VPs) are increasingly used because they are permanently available and fully scalable to a large audience. However, empirical studies comparing the differential effects of these assessment methods are lacking. Similarly, the relations between key variables associated with clinical reasoning performance in these assessment methods still require further research.

Objective:

We analyzed to what extent authenticity, cognitive load, and clinical reasoning (i.e., diagnostic accuracy and evidence generation) differ between the two assessment methods. Moreover, we examined how clinical reasoning performance is associated with professional knowledge, authenticity, and cognitive load.

Methods:

We conducted an experimental study with N=86 medical students (mean 26.03 years old, SD 4.71) focusing on history-taking in dyspnea cases. Conceptual and strategic knowledge were measured with a pretest. Participants then solved three cases with SPs and three cases with VPs in this repeated-measures study. After each case, students provided a diagnosis and rated authenticity and cognitive load. The provided diagnosis was scored in terms of diagnostic accuracy; the questions asked by the medical students were rated with respect to their quality of evidence generation.

Results:

Authenticity (2-tailed t81=11.12; P<.001) and the clinical reasoning variables of diagnostic accuracy (2-tailed t85=2.49; P=.01) and quantity of evidence generation (2-tailed t69=12.26, P<.001) were higher for SPs than for VPs. Cognitive load did not differ meaningfully between assessment methods (2-tailed t82=-0.38; P=.70). Clinical reasoning performance in the form of diagnostic accuracy in both assessment methods combined was positively associated with knowledge (r=.26, P=.008) and negatively correlated with cognitive load (r=-.31, P=.004). However, diagnostic accuracy did not correlate with authenticity (r=.05, P=.32) nor with the quality of evidence generation (r=.18, P=.053).

Conclusions:

The study corroborates the finding that conceptual and strategic knowledge are positively associated with clinical reasoning. The results on authenticity demonstrate that learners experience SPs as more authentic than VPs. However, the reported lack of a meaningful association between authenticity and diagnostic accuracy puts the importance of authenticity in assessment somewhat into perspective. Spending large amounts of the medical education budget on increasing the authenticity of assessment above a certain necessary threshold seems to bring few benefits. As high amounts of cognitive load are detrimental to performance in both assessment methods, cognitive load must be monitored and manipulated systematically. The clinical reasoning variable of diagnostic accuracy was higher for SPs than for VPs, which could potentially negatively affect students’ grades with VPs. This indicates that VPs must be adapted and customized in order to allow performance-based assessment without detrimental effects.