JMIR Publications

ABSTRACT

Background:

Analyzing nicotine and tobacco product (NTP) content on social media requires researchers to interpret rapidly evolving youth-oriented trends and platform norms. Traditional qualitative coding approaches that rely primarily on adult researchers often fail to capture these nuances and may misinterpret content intended for younger audiences. To address this methodological gap, we implemented a Vertically Integrated Project (VIP) model that engaged coders across multiple age groups in the parallel analysis of Zyn-related YouTube Shorts. Our research team consisted of high school (HS) students, undergraduate (UG) students, and public health faculty members. This approach allowed us to examine how coder demographics shape interpretation and to assess the feasibility and value of youth-inclusive coding frameworks for social media research.

Objective:

This study aims to examine how a VIP model could be applied to the qualitative analysis of social media content related to emerging NTPs.

Methods:

We manually extracted 300 YouTube Shorts containing #Zyn on September 18, 2024. Each team of two HS students, two UG students, and two public health faculty used an existing codebook to independently code the same 50 videos per round across six iterative rounds. After each round, inter-rater reliability (IRR) was calculated using Cohen’s κ and percent agreement as complementary indicators. Following completion of all rounds, all coders participated in a 90-minute Zoom debriefing session. The transcript was analyzed using a combined deductive-inductive thematic approach to identify challenges and insights related to collaborative coding within the VIP model.

Results:

IRR improved across six iterative rounds, with the UG team achieving the highest reliability in the final coding round (κ: 0.40-0.63). Several constructs showed consistently high percent agreement but low and variable κ values, particularly within the HS team. For individual codes, agreement on male stereotype was stronger among the more experienced teams (UG: κ=0.75; faculty: κ=0.66) than the HS team (κ=0.10). Debriefing findings indicated that disagreement reflected factors such as evolving code definitions, interpretive drift across rounds, and differences in cultural and experiential lenses. Coders noted that the codebook struggled to capture ambiguity in short-form content and that teams developed distinct heuristics for resolving uncertainty. Suggested improvements included rotating coding pairs and expanding codebook examples.

Conclusions:

This study demonstrates that applying a VIP model to the coding of social media content is both feasible and methodologically valuable. Rather than treating lower IRR as a failure, the model reframes interpretive divergence as an insight into how age, experience, and cultural background shape content interpretation. Integrating adolescents as analytic partners enhanced real-world relevance when examining youth-oriented digital content. Additionally, the VIP model supports educational and equity-related goals by fostering reciprocal learning across age groups and helping to build a sustainable, youth-engaged public health research pipeline.