JMIR Publications

ABSTRACT

Background:

The field of eHealth is growing rapidly and chaotically. Health care professionals (HCPs) need guidance on reviewing and assessing health-related smartphone applications (apps) to propose appropriate ones to their patients. To date, however, no framework or evaluation tool fulfills this purpose.

Objective:

Therefore, prior to developing a tool to help HCPs assess and recommend apps to their patients, we aimed to create an overview of published criteria to describe and evaluate health apps.

Methods:

1) We conducted a systematic review to identify existing criteria for eHealth smartphone app evaluation. Relevant databases and trial registers were queried for articles. Articles were included that a) described tools, guidelines, dimensions and/or criteria to evaluate apps; were b) available in full text; and c) written in English, French, German, Italian, Portuguese, or Spanish. 2) We proposed a conceptual framework for app evaluation based on the dimensions reported in the selected articles. This was revised iteratively in discussion rounds with international stakeholders. 3) The conceptual framework was used to synthesize the reported evaluation criteria. The list of criteria was discussed and refined by the research team.

Results:

1) Screening 1’258 titles yielded 128 articles that met the inclusion criteria. Of these, 30 used self-developed criteria and described their development processes incompletely. While 43 evaluation instruments were used only once, six were used in multiple studies. Few articles reported following theoretical guidelines (n=83); those that did note 37 theoretical frameworks. 2) Based on the selected articles, we propose a conceptual framework to explore six app evaluation dimensions: context, stakeholder involvement, features & requirements, development processes, implementation and evaluation. 3) After standardizing definitions, we identified 205 distinct criteria. By consensus, the research team re-labeled twelve of these and added eleven more — mainly relating to ethical, legal and social aspects — resulting in 216 evaluation criteria in total. No criteria had to be moved between dimensions.

Conclusions:

This study provides a comprehensive overview of criteria currently used in clinical practice to describe and evaluate apps. This is necessary because no reviewed criteria sets were inclusive, and none included consistent definitions/terminology. While the resulting overview is impractical for use in clinical practice in its current form, this confirms the need to craft it into a purpose-built, theory-driven tool. Therefore, in a subsequent step, based on our current criteria set, we plan to construct a two-part app evaluation tool: a short section (including one to three questions per dimension) to quickly disqualify clearly-unsuitable apps, and a longer one to investigate more likely candidates in closer detail. We will employ a Delphi consensus-building process and develop a user manual to prepare for this undertaking. Clinical Trial: PROSPERO CRD42021227064