JMIR Publications

ABSTRACT

Background:

Innovation in artificial intelligence has proliferated exponentially in recent years, including in potential applications to health and healthcare systems. Beyond technical performance, trust and perception influence how these technologies are received and used in real-world settings. Social media platforms like X (Twitter) offer valuable, real-time insights into public opinion and discourse, yet are underexplored in the context of trust in digital health and technology. Understanding these evolving perspectives is essential for guiding responsible AI development and policy.

Objective:

The objective this study was to evaluate and apply LLMs to the annotation and large-scale analysis of social media discussions on health AI, including measures of confidence in the technology, emotional reactions, and sentiment.

Methods:

We developed and benchmarked a sentiment analysis framework using GPT-3.5-Turbo to assess overall sentiment and confidence across six key domains of health AI: usefulness, safety, trust, privacy, ethics, and quality. Data were collected from X (Twitter) between January 1 and December 5, 2023. A total of 388,009 social media posts were analyzed for sentiment and AI confidence, with a subset of 268,347 posts further examined using Emollama-7b for emotion classification and Latent Dirichlet Allocation for thematic analysis. WHO regions were characterized in terms of sentiment, confidence, emotion, and discussion focus.

Results:

GPT-3.5-Turbo classification of sentiment and the six aspects of health AI confidence achieved wF1-scores > 0.60 using highest performing prompting techniques. Findings show that global sentiment toward health AI is predominantly positive (65.26%) and emotionally optimistic (83.62%), though marked by regional variation. The Eastern Mediterranean and South-East Asia regions exhibited the highest levels of positivity and confidence, often linked to themes of business and innovation. In contrast, the Western Pacific region expressed lower confidence and greater concern about AI-human alignment, while showing strong interest in health AI research. Privacy emerged as the most salient global concern, while discourse in the Americas was uniquely focused on algorithms and data usage.

Conclusions:

Our study demonstrates the feasibility and value of LLM-powered social listening for real-time, domain-specific sentiment analysis. This approach offers a scalable and adaptable tool for monitoring public trust, informing policy, and supporting the responsible governance of health AI technologies at a global scale.