JMIR Publications

ABSTRACT

Background:

The rapid adoption of generative AI in the community and healthcare necessitates reliable, human-led evaluation.

Objective:

Develop a standardised rubric to support healthcare professionals, administrators, and researchers in evaluating generative AI outputs in health contexts.

Methods:

A rubric was developed through a four-stage mixed-methods study: literature synthesis, expert refinement, pilot application, and post-pilot feedback. The rubric was informed by 72 publications and refined through iterative feedback from a multidisciplinary expert panel (n=23). The pilot rubric was applied across four healthcare scenarios: responses to health-related Reddit questions, generation of health-related images and videos, responses to health professional continuing education questions, and automated physician-to-physician clinical documentation generation. Across these scenarios, 15 clinicians (medical practitioners, pharmacists and medical oncologists) evaluated 250 human- and AI-authored input-output pairs using domain-level ratings and qualitative comments.

Results:

The final rubric, OMNI, comprised 10 evaluation domains (Accuracy, Safety, Recognises Uncertainty, Completeness, Communicates Clearly, Empathetic, Context Aware, Referencing, Follows Instructions, and Avoids Coerciveness) assessed on four-point Likert scales, supported by detailed instructional and example materials. Across the four pilot scenarios, assessors generated 9,600 domain-level ratings and 4,010 qualitative comments. Mean pairwise agreement for the four-point Likert groups ranged from 30.8% to 53.4%, increasing to 59.3% to 79.6% when collapsed into broad agreement categories; Gwet's AC2 ranged from 0.49 to 0.80. OMNI differentiated strengths and weaknesses across models and scenarios, with recurrent limitations in Referencing, Recognises Uncertainty, Completeness, Empathetic, and Accuracy identified across scenarios. Multimodal image and video outputs performed worst overall. Assessors reported OMNI was usable and indicated they would reuse it.

Conclusions:

OMNI is a standardised, accessible rubric for expert-led evaluation of generative AI outputs in health contexts. Pilot application demonstrated that OMNI could differentiate AI performance across models and domains while generating structured feedback.Further work is needed to assess its applicability to purpose-built health AI tools, agentic systems, and organisational evaluation settings.