JMIR Publications

ABSTRACT

Background:

In order for Patient safety incident reporting to support organizational learning, investigation reports must be complete, factual, objective and actionable. However, the conventional way in which reports are generated are time-consuming and variable in terms of completeness, especially in the aspects of possible contributing factors and follow-up actions. Conventionally, incident reports are drafted freehand by individuals who witnessed or directly involved in the incident, followed by edits and comments by their reporting supervisors. Project NARRATE (Nursing AI-Refined for Accurate Transcription of Events) is a nursing-led ambient artificial intelligence (AI) voice-to-note workflow designed to align with the World Health Organization (WHO) Minimal Information Model for Patient Safety Incident Reporting and Learning Systems (MIM PS).

Objective:

This study evaluated the effectiveness of NARRATE in enabling higher completeness and narrative quality than conventional reports for fall and medication administration-related incidents.

Methods:

We conducted a retrospective pre-post observational document-quality study at a tertiary academic medical center in Singapore. We reviewed 150 deidentified completed supervisor investigation reports: 75 conventional reports sampled from June–August 2025 (pre) and 75 NARRATE reports sampled from January–March 2026 (post). In the design of NARRATE, advanced WHO MIM PS–aligned prompts, Situation-Background-Assessment-Recommendation (SBAR) structure, and placeholders (when safety-critical elements were absent) were used to enable completeness of reports. Two blinded reviewers independently rated reports using an advanced WHO MIM PS–aligned completeness checklist (item-level content validity index 1.00) and an adapted Physician Documentation Quality Instrument-9 (PDQI-9) based narrative quality scale. Interrater reliability was assessed using 2-way random-effects absolute-agreement intraclass correlation coefficients. Between-group differences were tested using independent-samples t tests with Welch correction where appropriate, and Cohen d was reported.

Results:

All 150 reports were analyzed. Interrater reliability was acceptable for WHO total completeness (single-measure ICC 0.689, 95% CI 0.577–0.772; average-measure ICC 0.816, 95% CI 0.732–0.872) and strong for adapted PDQI-9 mean score (single-measure ICC 0.861, 95% CI 0.813–0.897; average-measure ICC 0.925, 95% CI 0.897–0.946). NARRATE-period reports had significantly higher WHO total completeness scores than conventional reports (mean 13.61, SD 2.54 vs mean 11.81, SD 3.39; mean difference 1.79, 95% CI 0.83–2.76; P<.001; d=0.60, 95% CI 0.27–0.93) and higher adapted PDQI-9 mean scores (mean 3.97, SD 0.23 vs mean 3.52, SD 0.55; mean difference 0.46, 95% CI 0.32–0.59; P<.001; d=1.07, 95% CI 0.73–1.41). Exploratory domain analyses showed gains in WHO explanation and actions domains and in adapted PDQI-9 thoroughness, usefulness, comprehensibility, synthesis, and fairness/balance.

Conclusions:

A nursing-led, AI-assisted structured narration workflow was associated with higher completeness and improved narrative quality of supervisor investigation reports. The cue helped to ensure convert omissions of information were minimised . Further prospective evaluation should assess downstream effects on action planning, organizational learning, and safety outcomes. Clinical Trial: Not applicable. This study was not a clinical trial and was not prospectively registered. The study received IRB exemption.