JMIR Publications

ABSTRACT

Background:

Medical documentation burden remains a significant driver of physician burnout1,2, particularly in high-volume environments like the pediatric emergency department. While ambient artificial intelligence (AI) scribing has shown promise in adult medicine settings in decreasing mental and documentation burden3,4, pediatric care involves unique triadic interactions between the physician, patient, and often several caregivers. This inherent situation might expose the AI software to an as-of-yet untested accuracy issue when a LLM is challenged to extract information from several patient sources.

Objective:

We conducted a 3-month pilot study at UCSF Benioff Children’s Hospital Oakland to evaluate the feasibility of this technology and early clinical perceptions by physicians in a dedicated pediatric urgent care setting. This was an exploratory effort meant to inform further hypotheses for a future formal prospective trial.

Methods:

We conducted a 3-month pilot study at UCSF Benioff Children’s Hospital Oakland to evaluate the feasibility of this technology and early clinical perceptions by physicians in a dedicated pediatric urgent care setting. This was an exploratory effort meant to inform further hypotheses for a future formal prospective trial.

Results:

Between April and August 2024, eight pediatric emergency physicians voluntarily utilized an AI scribe software (Ambience Healthcare, San Francisco, CA) after an hour training onboarding session. There are 32 providers in the group and 8 volunteered for the project. The AI scribing software generated drafts for the History of Present Illness, Review of Systems, and Physical Exam, which were then edited and accepted by the provider. At the time, Medical Decision Making was excluded from the AI-generated content. The software was used when the participants worked alone without assistance from trainees. Measured outcomes: physician perceptions of these interactions and their concomitant AI-generated notes were evaluated by four 5-point Likert-scored questions. These questions were asked both just before beginning the use of the software and then after three months of its use. As this was an exploratory feasibility study, results are presented descriptively. In the three months of this trial, there were 320 patient encounters performed by the eight providers. Mean scores showed modest upward trends across all six parameters, with no metrics declining. The most notable shifts occurred in perceived documentation manageability and the ability to complete charting within a shift. (see Table 1) Table 1: Mean Likert Scores of Physicians’ Perception (n=8) Question Before Using AI Scribe software After 3-month’s use of AI Scribe software My documentation is Manageable 2.25 2. Work Feels Sustainable 2.13 2.25 Time spent on Documentation is easy 1.00 1.26 I complete most of my charting on shift 1.0 1.25 Scoring: 1, strongly disagree 5, strongly agree

Conclusions:

While limited by small sample size and potential selection bias among early adopters, these findings suggest that ambient AI is deemed by clinicians to be a feasible tool for capturing and notating the complex, multi-party dialogue inherent in pediatric care. These data serve as a foundation for future powered trials building off of these perceptions. We now will perform a larger prospective trial in a study evaluating AI scribing’s utility in a ‘triadic’ clinical encounter environment of Pediatric Emergency Care, using metrics typical of ambient listening technology looked at in recent prospective trials. These will include time in notes, ‘pajama time’, patient throughput, edit frequency and relative value units generated Clinical Trial: https://irb.ucsf.edu/