JMIR Publications

ABSTRACT

Background:

In intervention studies, process evaluation measures the context in which an outcome was or was not achieved through ongoing monitoring of operations to document intervention dose, fidelity, facilitators, obstacles, etc. Mobile apps are a potentially less burdensome method to collect process metrics in real time from participants. While apps are increasingly research-driven, they are not always developed with attention to usability of features for the target user group. Usability testing uncovers gaps in researchers’, developers’, and users’ mental models of an efficient, effective, and satisfying product, to facilitate design improvement. Understanding user demographics also helps fulfill this goal.

Objective:

This study (1) describes the iterative development a mobile app for collecting process evaluation metrics in a participatory Total Worker Health® intervention with healthcare workers; (2) identifies the functions most used by end-users; and (3) compares app usage by demographic and job characteristics of app users.

Methods:

An app was developed for evaluation of the CPH-NEW Healthy Workplace Participatory Program in public sector healthcare institutions. Members of labor-management health and safety committees, program champions, and managers were the target app users. Four functions are built into the app to record: meetings created; project-related communications; time spent on project activities; chat messages; and post-meeting feedback surveys, which cover participant engagement, group dynamics, facilitator effectiveness, and usefulness of the intervention at each step. Screen time was recorded with Google Analytics. Pilot tests were conducted in stages to assess both functionality and usability across a combination of device software, hardware, and platforms. Students working within the research team were recruited to test the first fully functioning prototype and asked to provide qualitative and quantitative feedback while performing task scenarios expected from end-users. Feedback was used to fix reported issues and inform further development. In the second stage, the app was deployed to all end-users on site. End-user volunteers where requested to complete task scenarios and provide feedback as part of their onboarding process.

Results:

Functionality problems were fixed as they were documented. System Usability Scale scores for seven student testers corresponded to “good” usability (mobile app=72.9, website=72.5), whereas fifteen end users regarded the system as “acceptable” usability (mobile app=64.7, website=62.5). Participants aged 40-54 had almost double the amount of screen time than younger and older age groups (p=0.034). Users suggested useful features that would have result in large-scale restructuring of the backend; these were weighed for benefits versus the budget. Data collection was replicated in paper formats to capture all potential participants enrolled in this workplace intervention study.

Conclusions:

Mobile app development benefits from early iterative testing to find and fix problems or usability issues, which can assist in increasing usage of the product and prevent potential data loss.