JMIR Publications

ABSTRACT

Background:

Musculoskeletal disorders negatively impact millions of patients globally placing significant demand on healthcare systems. Digital technologies improving clinical outcomes and efficiency across the care pathway are development priorities, however rigorous evaluation must be completed to ensure efficacy. We have developed a musculoskeletal Digital Assessment Routing Tool (DART) enabling users to self-assess and be directed to right care, first time that requires usability evaluation prior to clinical trials.

Objective:

This study aimed to assess and resolve all serious DART usability issues, optimizing system performance to create a positive user experience and system adoption. In turn, providing justifiable confidence to progress to full scale randomized controlled trials when DART is integrated into musculoskeletal management pathways.

Methods:

An iterative-convergent mixed-methods design was used, with 22 adult participants using 50 different clinical presentations over 5 testing rounds across 4 DART iterations. Participants were recruited using purposive sampling with quotas for age, habitual internet use and English language ability. Quantitative data collection was defined by the constructs within the ISO 9241-210-2019 standard, with user satisfaction measured by the system usability scale. Study endpoints were resolution of all Grade 1 and 2 usability problems and a mean system usability scale score of 80 or over across a minimum of 3 user group sessions.

Results:

All participants (mean age 51.3 years, range 20-77 years) completed the study. Every assessment resulted in a recommendation with no DART system errors and a mean completion time of 5.2 minutes (SD 4.44, range 1-18). Usability problems reduced from 12 to zero, with trust and intention to act improving during the study. The relationship between electronic health literacy and age as explored with a scatterplot and calculation of a Pearson correlation coefficient was performed for all participants r=-0.2 (n=20) and repeated with a potential outlier removed r= -0.23, with no meaningful relationship observed or found for either. The mean satisfaction in daily internet users was highest (n, mean, 90% CL: 19, 86.5, ±1.78), compared with non-native English speakers (6, 78.1, ±9.17) and infrequent internet users scoring the lowest (3, 70.8, ±3.79), albeit the confidence intervals do overlap. The mean score across all groups was 84.3 (±4.67), corresponding to an “excellent” system, with qualitative data from all participants confirming DART was simple to use.

Conclusions:

All serious DART usability issues were resolved and a good level of satisfaction, trust, and willingness to act on the DART recommendation was demonstrated, thus allowing progression to randomized controlled trials assessing safety and effectiveness against usual care comparators. The iterative convergent mixed methods design proved highly effective in fully evaluating DART from a user perspective and could provide a blueprint for other researchers of mHealth systems.