JMIR Publications

ABSTRACT

Background:

Artificial intelligence (AI) is increasingly recognized as a transformative tool in healthcare, with the potential to enhance efficiency, diagnostic accuracy, and patient outcomes. However, while AI research often focuses on model development, there is limited knowledge on how to effectively implement and evaluate AI solutions in clinical practice. The Non-Adoption, Abandonment, Scale-up, Spread, and Sustainability (NASSS) framework has been used to assess the adoption of digital health technologies, but its applicability for real-time AI implementation evaluations remains underexplored. This study examines the first large-scale implementation of a commercial AI algorithm in Norwegian healthcare, using NASSS as an evaluation framework.

Objective:

The goal of the paper is to assess the usefulness of the NASSS framework for longitudinal process evaluations of AI implementations in healthcare, identifying key barriers, facilitators, and socio-technical complexities encountered during different implementation phases.

Methods:

A formative process evaluation was conducted between 2020-2024, following the implementation of the BoneView AI algorithm for fracture detection at Vestre Viken Health Trust. The study included 63 stakeholder interviews, observations, and document analyses at different phases of the process. The interview guide and thematic analysis were structured according to the seven NASSS categories, we also sorted the data in each category into 2-3 subtopics outline barriers and values generated at different phases of the implementation process.

Results:

The evaluation identified many critical factors for successful AI implementation. These include establishing clinician trust, integrating AI into clinical workflows, ensuring digital maturity, addressing legal and validation concerns, and optimizing organizational management strategies. Key challenges included radiographers' reluctance to take on additional responsibilities without proper training and compensation, , AI not being an “of the shelf” solution after all but in demand of extensive validation, the need to build digital maturity in the organization before implementing to advanced AI, and misalignment in financial models, where the imaging department bore the cost while the primary benefits were realized by emergency rooms and patients. The study also highlighted gaps in the NASSS framework when applied to AI implementation. Specifically, NASSS lacked dedicated categories for workflow evolution over time and changes in user expectations and attitudes, both of which were essential for evaluating AI adoption. To address these gaps, the study proposes two additional subcategories within NASSS to better capture these dynamics

Conclusions:

The NASSS framework proved valuable for evaluating AI implementation as a complex, socio-technical process, providing insights beyond traditional cost-benefit analyses. However, modifications are needed to improve its applicability for AI-specific challenges, including a stronger emphasis on workflow adaptation, evolving user perceptions, and longitudinal effects. The findings from this study contribute to a deeper understanding of AI deployment in healthcare and provide a foundation for refining evaluation frameworks to support future AI implementations.