JMIR Publications

ABSTRACT

Background:

Usability tests can be either formative (where the aim is to detect usability problems) or summative (where the aim is to benchmark usability). There are ample formative methods that can take into account user characteristics and contexts (i.e., cognitive walkthrough, co-discovery, verbal protocols). This is especially valuable for eHealth applications, as health conditions can influence user-technology interaction. However, most summative usability tests do not consider eHealth-specific factors that could potentially affect the usability of a system. One of the reasons for this, is that there are currently no fine-grained frameworks or models of usability factors that are unique for the eHealth domain.

Objective:

The goal of this study is to develop an ontology of usability problems, specifically for eHealth applications with patients as their primary end-user group.

Methods:

We analysed eight datasets, containing the results of eight formative usability tests of eHealth applications. These datasets contained 400 usability problems that were usable for analysis. Both, inductive and deductive coding was used to create the ontology. Six datasets were used to create the ontology; two datasets were used for the validation of the framework by assessing the inter-coder agreement.

Results:

Eight main categories of usability factors were identified: (1) System Basic Performance, (2) Task-Technology Fit, (3) User Accommodativeness, (4) Interface Design, (5) Navigation & Structure, (6) Information & Terminology, (7) Guidance & Support, (8) Satisfaction. These eight categories contain a total of 21 factors: 14 general usability factors and seven context-specific factors. Cohen’s kappa was calculated for two datasets, on both category and factor level. The Kappa’s were all between k = 0.62 and k = 0.67, which is acceptable. Descriptive analysis revealed that approximately 70% of the usability problems can be considered as general usability factors and 30% as eHealth-specific usability factors.

Conclusions:

Our ontology provides a detailed overview of usability factors for eHealth applications. Current usability benchmarking instruments include only a subset of the factors that emerged from our study and are therefore not fully suited for summative evaluations of eHealth applications. Our findings can support the development of new usability benchmarking tools for the eHealth domain.