JMIR Publications

ABSTRACT

Background:

Social skills training by human trainers is a well-established method of teaching appropriate social and communication skills and strengthening social self-efficacy. Specifically, human social skills training is a fundamental approach to teaching and learning the rules of social interaction. However, it is cost-ineffective and offers low accessibility, since professional trainers are limited. In our previous work, we attempted to automate social skills training by developing a conversational agent that taught social skills through interaction. Long-term validation of such an automated training system is needed.

Objective:

This study aims to validate the training effect of a social skills training system during a four-week randomized controlled pilot trial. We compare two groups (with and without training) and hypothesize that trained people will improve their social skills and overcome their social anxiety to a greater extent than the non-trained group. Furthermore, this study seeks to clarify the optimal sample size and effect size for future larger-scale evaluations.

Methods:

For the study, 26 Japanese participants are separated into the two groups, where we hypothesize that Group 1 (with training) will make greater improvement than Group 2 (without training). In addition to questionnaires, we conduct a Role-play Performance Test that requires the social cognition and expression of participants in new role-play scenarios. Blind ratings by third-party trainers are made by watching recorded role-play videos. A non-parametric Wilcoxson rank sum test is performed for each variable. Improvement between pre- and post-training evaluations are used to compare the two groups. Moreover, we compare the statistical significance from the questionnaires and ratings between the two groups.

Results:

Out of 26 recruited participants, 18 individuals completed this experiment: 9 people in Group 1 and 9 in Group 2. Those in Group 1 achieved significant improvement in generalized self-efficacy (P=0.02). We also found a significant decrease in state anxiety presence (P=0.04) measured by the State-Trait Anxiety Inventory. For ratings by third-party trainers, clarity of speech was significantly strengthened in Group 1 (P=0.03).

Conclusions:

Our findings reveal the usefulness of the social skills training system after a one-month training period. This study confirms a large effect size between groups, thus indicating the need for further larger-scale evaluations.