JMIR Publications

ABSTRACT

Background:

It is well known that recommendations from electronic medication alerts are seldom accepted or acted on by users. Key factors affecting the effectiveness of medication alerts include system usability and alert design. In response, human factors principles that apply knowledge of human capabilities and limitations are increasingly used in the design of health technology to improve usability of systems.

Objective:

We set out to develop an evidence-based tool that allows valid and reliable assessment of computerised medication alerting systems. This tool is intended to be used by hospital staff with detailed knowledge of their hospital’s computerised provider order entry (CPOE) system and alerts to identify and address potential system deficiencies.

Methods:

The Tool for Evaluating Medication Alerting Systems (TEMAS) was developed based on human factors design principles and consists of 66 items. Eighteen staff members from six hospitals used the TEMAS to assess their medication alerting system. Data collected from participant assessments were used to evaluate the validity, reliability, and usability of the TEMAS. Validity was assessed by comparing TEMAS results with prior in-house evaluations; Reliability was measured using Krippendorff’s alpha to determine agreement between assessors; and a short survey was used to determine usability.

Results:

Participants reported mostly negative (n=8/17) and neutral (n=7/17) perceptions of alerts in their medication alerting system. However, validity of the TEMAS was unable to be directly tested as participants were unaware of any results from prior in-house evaluations. Reliability of the TEMAS, as measured by Krippendorff’s alpha, was low to moderate (range: .26 - .46) but participant feedback suggest that an individual’s knowledge of the system varied depending on professional background. In terms of usability, the TEMAS items were generally easy to understand (61%), but participants suggested revisions to items (n=22) to improve clarity.

Conclusions:

This preliminary evaluation of the TEMAS allowed identification of components of the tool that required modification to improve usability and usefulness. It also revealed that to be effective in facilitating a comprehensive evaluation of a medication alerting system, the TEMAS should be completed by a team of multi-disciplinary hospital staff from both clinical and technical backgrounds in order to maximise knowledge of systems.