JMIR Publications

ABSTRACT

Background:

Consumer-oriented mobile self-diagnosis apps have been developed using undisclosed algorithms, presumably based on machine learning (ML) and other artificial intelligence (AI) technologies. The FDA now discerns apps with learning AI algorithms from those with stable ones and treats them as medical devices. No self-diagnosis app testing has been performed in the field of Ophthalmology so far.

Objective:

To test apps that were previously mentioned in scientific literature on a set of diagnoses in a deliberate time interval, comparing the results and looking for differences that hint to “non-locked” learning algorithms. A set of ophthalmologic diagnoses was used to simultaneously test their diagnostic efficiency and treatment recommendations in this specialty.

Methods:

Four apps from literature were chosen (Ada, Babylon, Buoy and Your.MD). Three ophthalmic diagnoses in three levels of urgency were used (Glaucoma, Retinal tear, Dry eye syndrome). Two years was chosen as a time interval between the tests (2018 and 2020).

Results:

Two apps (Ada, Buoy) asked significantly more questions than the other two (P=.00, respectively). The number of questions asked differed for none of the four apps between 2018 and 2020. For all four, the diagnostic efficacy and treatment recommendations differed strongly between 2018 and 2020, indicating “non-locked” learning algorithms using AI technologies. None of the apps provided correct diagnoses and treatment recommendations for all three diagnoses.

Conclusions:

Systematic studies on a wider scale are necessary for healthcare providers and patients to correctly assess the safety and efficacy of such apps and for correct classification by healthcare regulating authorities.