JMIR Publications

ABSTRACT

Background:

Words are a natural way of describing mental states for humans, while numerical values are a convenient and effective way to carry out quantitative psychological research. With the growing interest of researchers in gaming disorder, the number of screening tools is growing. Nevertheless, they all require self-quantification of mental states. The rapid development of natural language processing creates an opportunity to supplement the traditional rating scales with a Question-based Computational Language Assessment (QCLA) approach that gives a deeper understanding of the studied phenomenon without losing the rigor of quantitative data analysis.

Objective:

The aim of the study was to investigate whether the transformers-based language models analysis of text responses from active gamers is a potential supplement to traditional rating scales. We compared a tool consisting of four open-ended questions formulated based on the clinician's intuition (not directly related to any existing rating scales for measuring gaming disorders) with the results of one of the commonly used rating scales.

Methods:

Participants (N = 417) were requested to answer the Word Based Gaming Disorder Test, consisting of four open-ended questions regarding gaming. Subsequently, they completed a close-ended Gaming Disorder Test based on numerical scale. Following that, their responses to the open-ended questions were vectorized using HerBERT, a large language model based on Google's BERT. Lastly, a machine learning model, specifically ridge regression, was utilized to predict the scores from the Gaming Disorder Test based on the features from the vectorized open-ended responses.

Results:

The Pearson correlation between the observable Gaming Disorder test scores and the predictions made by the model was .476 when using all four respondents' answers as features. When using only one out of the four text responses, the correlation varied from .274 to .406.

Conclusions:

Short open-ended responses analyzed by natural language processing can contribute to a deeper understanding of gaming disorder at no additional cost in time. The obtained results confirmed two out of three pre-registered hypotheses. Written statements analyzed using the model correlate with the rating scale results. In addition, the inclusion in the model of data from more responses that take into account different perspectives on gaming improves the performance of the model. However, there is room for improvement - especially in terms of supplementing the questions with content that more directly corresponds to the definition of gaming disorder.