JMIR Publications

ABSTRACT

Background:

Background:

There are a huge number of health-related apps available, and the numbers are growing fast. However, many of them have been developed without any kind of quality control. In an attempt to contribute to the development of high quality apps and enable existing ones to be assessed, several guides have been developed.

Objective:

Objective:

The main aim of this study was to study the interrater reliability of a new guide – the Mobile App Development and Assessment Guide (MAG) – and compare it with one of the most used guides in the field, the Mobile App Rating Scale (MARS).

Methods:

Methods:

In order to study the interrater reliability of MAG and MARS, we evaluated the four most downloaded chronic health apps for Android and IOS devices. A group of eight reviewers, which included different types of stakeholders such as clinical researchers, engineers, health-care professionals, and end-users as potential patients, independently evaluated the quality of the apps using MAG and the MARS. We used Krippendorff's alpha to calculate the interrater reliability.

Results:

Results:

Only a few categories of MAG and MARS demonstrated a high interrater reliability. Although MAG was found to be superior, there was considerable variation in the scores between the different types of reviewer. The categories with the highest interrater reliability in MAG were “Security” (alpha = 0.78) and “Privacy” (alpha = 0.73).

Conclusions:

Conclusions:

This study shows that some categories of MAG have significant interrater reliability and that the MAG scores are better than the MARS scores. However, there is great variability in the responses, which seems to be associated with subjective interpretation by the reviewers.