JMIR Publications

ABSTRACT

Background:

Assessment of technical aptitude, cognitive abilities and personality characteristics is important in selecting candidates for surgical training. Currently, the selection of surgical training candidates is based on ineffective methods which have been shown to be poorly correlated with later performance.

Objective:

The present study examined the validity of two novel assessment methods developed for this purpose: a virtual reality (VR) technical aptitude test and a game-based assessment of cognitive abilities and personality characteristics.

Methods:

This study focused on relationships with other variables, one of the sources of validity evidence. The study had three phases. In Phase 1, we evaluated convergent and discriminant evidence of validity by assessing the correlation between interns’ performance in the two novel selection tests and in four traditional tests for assessing dexterity, visuospatial ability, intelligence, and personality. In Phase 2, we evaluated evidence for test–criterion relationship by assessing the correlation between residents’ performance in the two selection tests and their residency performance evaluations. In this phase we also assessed evidence for the fairness of the tests across genders. In Phase 3, we evaluated evidence for relationship with training level by administering the technical aptitude test to a sample of expert surgeons and comparing their performance with that of the residents and interns from the previous phases.

Results:

Interns’ scores on the novel selection tests were correlated with scores on the relevant traditional tests, providing convergent and discriminant evidence (Phase 1). Residents’ scores on the novel tests were significantly correlated with relevant performance criteria (Phase 2). In addition, no evidence for gender bias in the tests was found. Finally, based on data collected in all three phases, we found evidence for expert¬–novice differences, such that the technical aptitude test scores were correlated with surgical experience.

Conclusions:

The findings provide validity evidence supporting use of the novel VR and gamification tests for assessment of technical aptitude, cognitive abilities and personality characteristics in selecting candidates for surgical training. The evidence suggests that use of the tests may improve the selection of surgical residents.