JMIR Publications

ABSTRACT

Background:

While health apps are popular for both professional as well as non-professional users, they are often used without adequately taking aspects related to their quality under con-sideration. This may partially be due to inadequate awareness about which criteria to ap-ply, and how to prioritize them when evaluating an app.

Objective:

The study was designed to determine how physicians perceive a given set of app quality principles, and to develop a method for their prioritization based on these perceptions. In today’s busy medical field, this may be relevant for professionals who want to reduce time and effort when independently assessing apps, e.g. by applying only the most relevant quality principles.

Methods:

9503 members of two German professional societies in the field of orthopedics were in-vited by email to participate in an anonymous online survey (1-month period). Participants were asked to rate a set of 9 app quality principles using a Kano survey with so-called functional and dysfunctional (i.e., positively and negatively worded) questions. The data was then evaluated using methods based on Kano’s work, including a self-designed ap-proach.

Results:

N=382 participants completed the parts of the survey relevant to the work presented here (return rate of approx. 4%). These were randomly assigned to either of two groups (test group A, validation group B, both n=191). There were no statistically significant differences between both samples with respect to demographics (P .05 for all factors). Common evaluation strategies for Kano surveys did not lead to a conclusive prioritization of the principles, as they were rated too similar. This remained true even with a more elaborate approach of satisfaction and dissatisfaction indices (Timko’s approach), which, despite reflecting customer satisfaction or dissatisfaction in terms of two numerical values, did not allow for the desired ranking. Therefore, based on the aforementioned approach, an enhanced, so-called “In-Line-of-Sight” method was developed and applied. Using it, a point of view from either of the four categories allowed by Timko is taken, represented by a ranking coefficient. While, apart from resource efficiency, all other principles were previously simply assigned to the must-be category, the new method, applied from the must-be point of view, was able identify identical rankings for both groups: 1. legal conformity, 2. content validity, 3. risk adequacy, 4. practicality, 5. ethical soundness, 6. usability, 7. transparency, 8. technical adequacy, 1. resource efficiency, and finally, 9. resource efficiency.

Conclusions:

N=382 participants completed the parts of the survey relevant to the work presented here (return rate of approx. 4%). These were randomly assigned to either of two groups (test group A, validation group B, both n=191). There were no statistically significant differences between both samples with respect to demographics (P .05 for all factors). Common evaluation strategies for Kano surveys did not lead to a conclusive prioritization of the principles, as they were rated too similar. This remained true even with a more elaborate approach of satisfaction and dissatisfaction indices (Timko’s approach), which, despite reflecting customer satisfaction or dissatisfaction in terms of two numerical values, did not allow for the desired ranking. Therefore, based on the aforementioned approach, an enhanced, so-called “In-Line-of-Sight” method was developed and applied. Using it, a point of view from either of the four categories allowed by Timko is taken, represented by a ranking coefficient. While, apart from resource efficiency, all other principles were previously simply assigned to the must-be category, the new method, applied from the must-be point of view, was able identify identical rankings for both groups: 1. legal conformity, 2. content validity, 3. risk adequacy, 4. practicality, 5. ethical soundness, 6. usability, 7. transparency, 8. technical adequacy, 1. resource efficiency, and finally, 9. resource efficiency.