JMIR Publications

ABSTRACT

Background:

The proliferation of generative AI, like ChatGPT, has added complexity to the online environment by increasing the presence of AI-generated content (AIGC). Although social media platforms such as TikTok have begun labeling AIGC to help users distinguish it from human-generated content (HGC), little research exists on the effectiveness of these labels.

Objective:

This study investigates the impact of AIGC labels on user perceptions through a web-based experimental framework, aiming to refine the strategic use of such labels.

Methods:

We conducted a 2×2×2 mixed factorial experiment, using the presence of AIGC labels as the between-subjects factor, and information accuracy and content type (for-profit or non-profit) as within-subjects factors. The study employs a randomized controlled trial design. It is not a clinical trial involving patient recruitment, but a web-based experiment measuring the effectiveness of internet governance measures; thus, registration for clinical trials is not required. Participants, recruited via the Credamo data platform, were randomly assigned to either an experimental group (with labels) or a control group (without labels). Each participant evaluated four sets of content, providing feedback on perceived accuracy, credibility, and sharing willingness. Statistical analyses were performed using SPSS, employing repeated measures ANOVA and simple effects analysis, with significance set at P < .05.

Results:

This study recruited a total of 957 participants, and after screening, 400 participants each were allocated to the experimental and control groups, totaling 800 participants. The main effects of AIGC labels were not significant for perceived accuracy, credibility of information, or willingness to share. However, the main effects of information type were significant for all three dependent variables (P < .001), as were the effects of content type (P < .001). There were notable differences in interaction effects among the three variables. For perceived accuracy, the interaction between information type and content type was significant (P = .005). For information credibility, the interaction between information type and content type was significant (P < .001). Regarding willingness to share, both the interaction between information type and content type (P < .001) and the interaction between information type and AIGC labels (P = .008) were significant .

Conclusions:

This study found that AIGC labels do not significantly impact users' perceived accuracy, credibility of information, or willingness to share, suggesting that AIGC labeling is a viable governance strategy. However, the specific application of AIGC labels should be tailored based on the type of information and content category. For inaccurate information, AIGC labels can somewhat increase people's willingness to share, slightly improve people's perceived accuracy and promote the message credibility, although this effect is not statistically significant. This finding highlights the need for more detailed AIGC labeling strategies, which warrants further research in the future.