Accepted for/Published in: Journal of Medical Internet Research
Date Submitted: Dec 5, 2024
Date Accepted: Apr 8, 2025
Developing balanced quality indicators for monitoring virtual care in Ambulatory Care Environments: A Modified Delphi Panel Process
ABSTRACT
Background:
While the volumes of virtual visits accelerating in uptake following the COVID-19 pandemic, less is understood regarding the quality of virtual visits. Furthermore, there is limited guidance for decision makers on how best to evaluate the quality of virtual care initiatives. There is a clear need for a comprehensive set of relevant quality indicators for implementation teams and organizations interested in either evaluating or implementing a virtual care initiative.
Objective:
The objective of this study was to curate a set of ambulatory quality indicators grounded in the quintuple aim quality framework and National Academy of Medicine domains of quality, including sustainability to provide a broad view of quality for virtual care in ambulatory environments.
Methods:
Methods:
A three round modified Delphi panel process was conducted. Phase 1 involved curating a list of ambulatory virtual care quality indicators from published and grey literature along with stakeholder derived indicators generated through a pre-Delphi survey; these indicators were mapped and categorized to the Quintuple Aim and National Academy of Medicine (NAM) Quality Domains.. Phase 2 involved rating and ranking of the quality indicators. A 19-member pan-Canadian Delphi panel was convened. The panel included representation from patients, clinicians, policy-makers, academics, and administrators. Panelists rated indicators using the Agency for Healthcare Research and Quality measure attributes on 9-point Likert scale in round 1, followed by a virtual Delphi panel meeting to discuss indicators before round 2 of re-rating, before a final round 3 of ranking assessing importance of indicators within each quality domain and sub-domain. Novel quality indicators of virtual care to address gaps in literature were included through panel discussions, patient group consultations, and a pre-Delphi survey. For indicator advancement during rating exercises, 75% or more of panelist’s responses in the top tertile (between 7-9) with a median composite score of 7 or greater was required.
Results:
Results:
140 quality indicators were included and. progressed through the 3 Delphi rounds. There was minimal attrition among Delphi panel members (17/19 participated in all 3 rounds). After round 3, 27 indicators were compiled in the final scorecard. Aligned with the quintuple aim, these indicators include 14 centered on patient experience, 7 regarding provider experience, 2 focused on population health, 2 focused on health equity, and 2 focused on health system costs.
Conclusions:
Conclusions:
By creating set of comprehensive set of quality indicators of ambulatory virtual care relevant to stakeholders across the health system, comprehensive evaluations of virtual care programs can be conducted examining all relevant domains of quality. Using the balanced scorecard as a reference, organizations and institutions can compare projects across jurisdictions, and identify priority areas of quality to ensure their virtual care initiative is appropriate for the care being provided. Clinical Trial: This study was not registered
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.