JMIR Publications

ABSTRACT

Background:

Most U.S. health systems operate on a regional scale and face substantial financial and staffing pressures, intensified by physician burnout and difficulties in recruitment and retention. Ambient artificial intelligence (AI) documentation solutions have the potential to reduce burden and improve satisfaction, but vendor selection is often undermined by cognitive bias, unvalidated marketing claims, and limited real-world testing.

Objective:

To address this, McLeod Health developed and implemented an objective, multi-phase approach to evaluate and adopt an ambient AI solution across its multi-hospital system.

Methods:

Our evaluation process began in spring 2024 with four leading vendors tested through live clinical simulations using 15 complex outpatient scripts with organizational leaders serving as standardized patients. Ambient patient encounters were captured in real time, and AI-generated notes were scored by physicians, revenue cycle experts, and non-clinical reviewers for accuracy, billing quality, and readability. The top two vendors advanced to demonstrations of Epic workflow integration, with physician usability feedback guiding the final selection. In the third phase, the chosen vendor underwent a 90-day pilot across five ambulatory specialties beginning in September 2024, followed by system-wide implementation in January 2025. Key performance indicators included documentation time, coding and financial trends, as well as patient and provider satisfaction. All statistical comparisons were two-sided using 95% confidence intervals.

Results:

The three-phase evaluation process resulted in careful vendor selection. The pilot showed a 35.4% decrease in pajama time (P=.054) and a 28.3% decrease in time in note (P<.001). Coding patterns shifted toward higher-complexity visits, with a 3.8% increase in level 4 established patient visits (P=.05) and established patient volumes increasing by 8.5%, producing an estimated net revenue gain of $2,629 per provider per month. Patient satisfaction improved significantly across multiple domains, with large gains in listening, trust, communication and treatment information (all P<.001), exceeding gains from prior system-wide patient satisfaction initiatives. System-wide rollout has achieved 81% adoption, with more than 150,000 notes generated.

Conclusions:

Our structured, multi-phase evaluation process minimized vendor influence and cognitive bias during selection, validated results through real-world clinical testing, and enabled a successful system-wide rollout. This approach provides a practical framework for non-academic health systems to objectively assess, implement, and scale ambient AI solutions while preserving fairness, transparency, and measurable value.