JMIR Publications

ABSTRACT

Background:

The integration of Artificial Intelligence in healthcare presents as a significant opportunity to revolutionize patient care. In the United States, an estimated 129 million people have at least one chronic illness, with 42% having two or more. Despite being largely preventable, the prevalence of chronic illness is expected to rise and impose significant economic burdens and financial toxicity on healthcare.

Objective:

We leverage an interdisciplinary team encompassing nursing, public health, and computer science to optimize health through prevention education for cardiovascular and metabolic comorbidities in persons living with HIV. In this paper, we describe the iterative development of an intersectionality-informed large language model designed to support cardiometabolic health in this population.

Methods:

First, we curated data by scraping publicly available, authoritative, evidence-based sources to capture a comprehensive dataset, supplemented by publicly available HIV forum content. Second, we benchmarked candidate large language models and generated a fine‐tuning dataset using GPT-4 through multi-turn question–answer conversations, employing standardized metrics to assess baseline model performance. Third, we iteratively refined the selected model via Low-Rank Adaptation and reinforcement learning, integrating quantitative metrics with qualitative expert evaluations.

Results:

Pre-existing LLM models demonstrated poor n-gram agreement, dissonance from model answers (Accuracy 4.16, Readability 4.63, Professionalism 4.58), and difficult readability (Kincaid 8.54, Jargon 4.44). After prompt adjustments and fine-tuning, preliminary results demonstrate the potential of a customized LLaMA-based LLM to provide personalized, culturally salient patient education.

Conclusions:

In this study, we described the steps in developing an LLM using an interdisciplinary team. Through data collection, data scraping, model benchmarking, and model fine-tuning, our LLM’s performance improved substantially (Accuracy 5.0, Readability 4.98, Professionalism 4.98, Kincaid 7.17, Jargon 2.92). This demonstrates strong promise of model success post-finalization. In building an LLM for cardiovascular health promotion and patient education, this research contributes to innovative strategies for managing comorbid conditions in persons with HIV. Clinical Trial: N/A