Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Generative Artificial Intelligence for Rapid Implementation and Support of Evaluations (GenAI-RiSE): Leveraging Generative AI for Rapid Qualitative Analysis in Evaluating Veterans Health Administration Initiatives
ABSTRACT
Background:
Qualitative methods are often used to gather information and understand contextual factors and perceptions influencing implementation. Traditional methods are resource intensive. Rapid qualitative methods have been successfully utilized to efficiently analyze qualitative data particularly to support implementation efforts, where early and frequent qualitative data are often used to inform and adapt programs during the early adoption phases. Generative artificial intelligence (GenAI) has increasingly been used in health care settings for text summarization.
Objective:
The aim of this work was to assess the accuracy and utility of (GenAI) for rapid qualitative analysis, using two real-world pilot implementations.
Methods:
The United States Department of Veterans Affairs (VA) integrated VA GPT, a GenAI tool, designed to handle sensitive data behind the VA firewall. Integrating VA GPT, we developed Generative Artificial Intelligence for Rapid Implementation and Support of Evaluations (GenAI-RiSE) based on an established rapid qualitative analysis process to evaluate two independent qualitative datasets. Qualitative datasets included structured interviews that were recorded and transcribed using Microsoft Teams (Pilot A) and open-ended survey responses via Microsoft Forms (Pilot B). GenAI-RiSE for both pilots involved: 1) Study Preparation (human review), Data Organization (human review) and Data Summarization (GenAI with human validation). We evaluated the accuracy and utility of the GenAI-assisted process for organizing, summarizing, and analyzing the qualitative data output.
Results:
From April- May 2025 we applied GenAI-RiSE to 12 interviews (PilotA) and 20 surveys (Pilot B) for analysis. VA GPT was successfully integrated into rapid qualitative analyses, summarizing the data into key themes, producing effective insights and summaries of open-ended survey responses and structured qualitative interviews. No major errors of omission or commission (hallucinations) were identified once the final prompt selection was confirmed. Overall, GenAI did a better job at analyzing the non-complex data of the structured dataset vs. survey dataset. This novel method delivered timely, accurate, and validated outputs to enable data-informed leadership decision making impacting implementation efforts.
Conclusions:
GenAI-assisted qualitative methods like GenAI-RiSE can be used to support and improve qualitative data organization, summarization, and analysis. However, trained qualitative researchers and content experts are needed for context, nuances within analyses, and accuracy.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.