JMIR Publications

ABSTRACT

Background:

Few gamified cognitive tasks are subjected to rigorous examination of psychometric properties, despite their use in experimental and clinical settings. Even small manipulations to cognitive tasks require extensive research to understand their effects.

Objective:

With this study, we seek to research how game elements can affect the reliability of scores on a Stroop task. We specifically investigate within and across session performance consistency.

Methods:

We created two versions of the Stroop task, with and without game elements, and then tested each task with participants at two time points. In this paper, we report on the reliability of the gamified Stroop task, in terms of internal consistency and test-retest reliability, compared to the control task. We also compare the reliability of scores on a trial-by-trial basis, with consideration of the different trial types.

Results:

In session 1, the Stroop effect was reduced in the game version, indicating an increase in performance. Further, the game led to higher measures of internal consistency in both sessions for error rates as well as reaction times, which indicates a more consistent response pattern. Test-retest reliability analysis revealed a distinctive performance pattern depending on the trial type that may be reflective of motivational differences between the task versions. In short, especially in the incongruent trials where cognitive conflict occurs, the performance in the game reaches peak consistency after 100 trials, while performance consistency drops after 50 trials for the basic version and only catches up to the game after 250 trials.

Conclusions:

Even subtle gamification can impact task performance, albeit not only in a direct difference in performance between conditions. People playing the game reach peak performance sooner, and their performance is more consistent within and across sessions. We advocate for a closer look at the impact of game elements on performance. Further, given the increased reliability of game-like tasks, they may be especially suitable for assessing populations that are not able to perform the task for an extended time period.