JMIR Publications

ABSTRACT

Existing exergames predominantly adjust difficulty based on explicit performance metrics, neglecting the player's implicit affective state, which often leads to early disengagement due to boredom or frustration. To address this gap, we present a novel exergame system that implements emotion-in-the-loop dynamic difficulty adjustment (DDA) through real-time multimodal affective computing. The system integrates three core modules: (1) a computer-vision-based motion evaluation pipeline that computes a seven-dimensional similarity vector between the player's skeleton and a reference motion; (2) a multimodal emotion recognition module that fuses facial action units and deep facial features with motion-compensated heart rate variability (HRV) features via an adaptive entropy-weighted decision-level fusion strategy; and (3) a closed-loop difficulty controller that jointly reasons about performance and emotional valence to maintain the player in an optimal challenge zone. We conducted a controlled between-subjects experiment (N=20) comparing our adaptive system against a fixed-difficulty baseline during a 15-minute dance-exergame session. The proposed emotion recognition module achieved 81.07% accuracy (Macro-F1: 81.02%) in classifying seven emotional states under physical activity. More importantly, the adaptive group demonstrated significantly lower cognitive load (NASA-TLX, p<0.05), higher positive affect (PANAS, p<0.01), greater system usability (SUS, p<0.05), improved motion accuracy (82.5% vs. 74.8%, p<0.05), and a physiologically healthier training load as indicated by higher HRV RMSSD (34.6 ms vs. 24.3 ms, p<0.05). These results validate that incorporating affective sensing into the difficulty control loop not only enhances subjective user experience but also yields measurable improvements in training performance and physiological regulation, offering a promising pathway toward genuinely personalized and engaging exergame-based interventions.