JMIR Publications

ABSTRACT

Background:

Multiple-choice questions (MCQ) are an integral part of medical assessment. While post-hoc item analysis is the standard for quality assurance, little is known about how medical students engage with MCQ items.

Objective:

Using eye-tracking methodology, this study maps students’ cognitive, physiological and behavioral reactions during MCQ test taking to different test item properties.

Methods:

A total of 28 second- and eighth-semester medical students completed ten single-choice medical questions each while their gaze behavior was recorded using head-mounted eye-tracking devices. Questions were drawn from a standardized Progress Test with pre-known quality indicators. We analyzed the relationships between different item properties (difficulty, discrimination, position) and eye-tracking metrics (fixation count, fixation duration, blink count, pupil diameter) using linear mixed-effects models.

Results:

The strongest effects were induced by item difficulty that led to higher fixation counts (β = 3.80), longer fixation durations (β = 1.30), and higher blink counts (β = 2.80), consistent with higher cognitive load. Conversely, items with high discrimination indices (i.e., those that effectively separate high from low performers) showed shorter fixation durations (β = –0.60), suggesting faster processing and clearer information decoding cues. We observed that the first test item induced approximately three times more blinks than subsequent items, indicating an initial adaptation phase to test taking. Regarding student characteristics, eight semester students tended to show slightly larger pupil diameters than second semester students (β = 0.22), potentially reflecting higher task engagement.

Conclusions:

Distinct eye tracking patterns seem to reflect test properties and student characteristics. These findings suggest the potential of eye-tracking methodology as a complementary, proof-of-concept add-on for the process validation of medical assessments.