JMIR Publications

ABSTRACT

Background:

As TikTok becomes a growing source of health information, Arabic-language content remains largely unexamined. Cancer misinformation and lack of accessible, culturally relevant content may contribute to disparities in health knowledge, behaviors, and outcomes. AI tools such as large language models (LLMs) offer scalable solutions for content analysis, yet their utility in Arabic health communication remains underexplored.

Objective:

To characterize and evaluate the quality of Arabic-language TikTok videos on cancer prevention and explore the use of LLMs for scalable content analysis.

Methods:

We used the TikTok Research Application Program Interface (API) and a Generative Pre-trained Transformer (GPT) assisted keyword strategy to collect 1,800 Arabic-language TikTok videos (2021–2024). After transcription and preprocessing, the top 25% most-viewed videos (n=30) were manually coded for content type, cancer type, uploader identity, tone, scientific citation, and disclaimers. Video quality was assessed using Patient Education Materials Assessment Tool for Audiovisual Materials (PEMAT AV) for understandability and actionability, and the Global Quality Scale (GQS). GPT-4 was used to generate AI annotations, which were compared to human coding for select variables.

Results:

From an initial pool of 320 Arabic-language TikTok videos on cancer prevention, 30 top viewed videos were analyzed. Together, these videos amassed a total of 21.6 million views. Diet and alternative therapies were most common (50%) which included recommendations to reduce hydrogenated oils, increase fruit and vegetable intake, and the use of traditional remedies such as garlic and black seed. Only 6.6% of videos cited scientific literature. General cancer (53%), breast (17%), and cervical (14%) cancers were most frequently mentioned. Doctors led 30% of videos and were more likely to produce higher quality content, including significantly higher Global Quality Scores (median GQS = 4 vs 3, p = .06). Over half of the videos had low understandability (53%) and actionability (60%). Emotionally framed content had the highest engagement across likes and shares, although this did not reach statistical significance (p = .08 and p = .05, respectively). However, emotional tone was significantly associated with lower GQS scores (p = .01). GPT-4 showed high agreement with human coders for cancer type (κ = 1.0), strong agreement for GQS (κ = 0.94), but low agreement for tone classification (κ = 0.15), due to misclassification of emotional delivery from text only input.

Conclusions:

Arabic TikTok content on cancer prevention is highly engaging but varies in quality. AI assisted tools show strong potential for scalable, multilingual health content analysis, but limitations in interpreting more nuanced and audio-visual features such as tone remain.