JMIR Publications

ABSTRACT

Background:

Labeling color fundus photos (CFP) is an important step in the development of artificial intelligence (AI) screening algorithms for the detection of diabetic retinopathy. Most studies use the International Classification of Diabetic Retinopathy (ICDR) to assign labels to CFP, plus the presence or absence of macular edema. Images can be grouped as referrable or non-referrable according to these classifications. There is little guidance in the literature about how to collect and use metadata as a part of the CFP labeling process.

Objective:

This project was conducted to improve the quality of the Multimodal Database of Retinal Images in Africa (MoDRIA) by determining whether the availability of metadata during the image labeling process influences the accuracy, sensitivity, and specificity of image labels. MoDRIA was developed as one of the inaugural research projects of the Mbarara University Data Science Research Hub (MUDSReH), part of the Data Science for Health Discovery and Innovation in Africa (DS-I Africa) initiative.

Methods:

This is a crossover assessment with 2 groups and 2 phases. Each group had 10 randomly assigned labelers who provided an ICDR score and presence or absence of macular edema for each of 50 CRF in a test image with and without metadata including blood pressure, visual acuity, glucose and medical history. Specificity and sensitivity of referable retinopathy was based on ICDR scores and macular edema calculated using 2-sided T-test. Comparison of sensitivity and specificity for ICDR scores and macular edema with and without metadata for each participant was calculated using the signed rank test. Statistical significance was set at P<0.05.

Results:

The sensitivity for identifying referrable diabetic retinopathy with metadata was 92.8% (95% CI: 87.6-98.0) compared with 93.3% (95% CI: 87.6-98.9) without metadata, and the specificity was 84.9% (95% CI: 75.1-94.6) with metadata compared with 88.2% (95% CI: 79.5-96.8) without metadata. The sensitivity for identifying the presence of macular edema was 64.3% (95% CI: 57.6-71.0) with metadata, compared with 63.1% (95% CI: 53.4-73.0) without metadata, and the specificity was 86.5% (95% CI: 81.4-91.5) with metadata compared with 87.7% (95% CI: 83.9-91.5) without metadata. Sensitivity and specificity of ICDR score and presence or absence of ME were calculated for each labeler with and without metadata. No findings were statistically significant.

Conclusions:

The sensitivity and specificity scores for the detection of referrable diabetic retinopathy was slightly better without metadata, but the difference was not statistically significant. We cannot make definitive conclusions about the impact of metadata on the sensitivity and specificity of image labels in our study. Given the importance of metadata in clinical situations, we believe that metadata may benefit labeling quality. A more rigorous study to determine the sensitivity and specificity of CFP labels with and without metadata is recommended.