JMIR Publications

ABSTRACT

Background:

Virtual Patient Simulator is a tool that may generate a multi-dimensional representation of the student’s medical knowledge by analyzing the recordings of the user’s actions during a clinical simulation. Adequate metrics may provide teachers with valuable learning information.

Objective:

To describe the analytic metrics we used to analyze the clinical diagnostic reasoning of medical students obtained by a novel Cognitive Tutor and Simulator named Hepius embedding Natural Language Processing (NLP) techniques.

Methods:

Two clinical case simulations (Tests) were created to tune our metrics. During each simulation, students’ actions were logged into the program data base for off-line analysis. Twenty-six students, attending the 5th year of the School of Medicine at Humanitas University, underwent Test 1 (April 12th 2019) which simulated a patient suffering from dyspnea. Test 2 (May 21st 2019) dealt with abdominal pain and was attended by 36 students. Overall students’ performance was split into 7 issues: 1) the identification of relevant information in the given clinical scenario (SC); 2) history taking (AN); 3) physical exam (PE); 4) medical tests (MT) ordering; 5) diagnostic hypotheses (HY) setting; 6) binary analysis fulfillment (BA); 7) final diagnosis (RS) setting. Sensitivity (percentage of relevant information found) and precision (percentage of correct actions performed) metrics were computed for each issue and combined into a harmonic (F1), thereby obtaining a single score (1= maximal sensitivity and precision) evaluating the student’s performances. The seven F1-metric scores were further combined to obtained a convenient index assessing the student’s overall performances.The seven metrics were further grouped to reflect the student’s capability to collect (SC, AN, PE and MT) and to analyze (HY, BA and RS) information. A methodological score was computed on the basis of the discordance between the diagnostic pathway followed by the student and a reference one, previously defined by the teacher.

Results:

Mean overall scores were consistent between the two tests (0.6.±0.05 for Test 1 and 0.5±0.05 for Test 2). For each student, overall performance was achieved by a different contribution in collecting and analyzing information. Methodological scores highlighted some discordance between the reference diagnostic pattern previously set by the teachear and the one pursued by the student.

Conclusions:

Different components of the student’s diagnostic process may be disentangled and quantified by appropriate metrics applied on students’ actions recorded while addressing a virtual case. Such an approach may help teachers in giving students individualized feedbacks aimed at filling up knowledge drawbacks and methodological inconsistencies.