JMIR Publications

ABSTRACT

Background:

Objective adherence monitoring is a major unmet need in mobile and connected health. Accelerometer-based smart eye drop bottles can classify instillation-like bottle motion, but a positive motion event does not prove that fluid reached the ocular surface. This dose-delivery confirmation gap is especially important for adherence monitoring because a sensor-positive event without actual delivery can provide false reassurance.

Objective:

We aimed to quantify agreement between artificial intelligence/sensor-classified eye drop instillation events and an objective fluorescein-based reference standard for ocular-surface delivery, emphasizing the false-assurance rate, defined as sensor-positive but delivery-absent events.

Methods:

In this prospective cross-sectional validation study, 100 ophthalmology outpatients performed one unaided instillation from a sensor-equipped 0.2% fluorescein bottle. Participants were assigned to bilateral, right-eye, or left-eye instillation. Blue-light slit-lamp photographs obtained immediately before and within 30 seconds after instillation served as the reference standard. Two ophthalmologists independently graded delivery while masked to the sensor output and to each other. The artificial intelligence/sensor classification was generated independently and masked to the photographic result. Each participant contributed one paired observation. Overall agreement, sensitivity, positive predictive value, and the false-assurance rate were calculated with exact 95% confidence intervals; bootstrap and Jeffreys-prior Bayesian analyses were prespecified as auxiliary uncertainty analyses. Reporting followed STARD 2015.

Results:

The two graders agreed for all 100 participants (Cohen kappa=1.00). The validation matrix included 98 sensor-positive/delivery-confirmed, 1 sensor-negative/delivery-confirmed, 1 sensor-positive/delivery-absent, and 0 sensor-negative/delivery-absent observations. Overall agreement was 98.0% (95% CI 93.0%-99.8%). Sensitivity was 99.0% (98/99; 95% CI 94.5%-100.0%), and positive predictive value was 99.0% (98/99; 95% CI 94.5%-100.0%). The false-assurance rate was 1.0% (1/99; 95% CI 0.03%-5.5%). The bootstrap 95% percentile interval for the false-assurance rate was 0.0%-3.1%, and the posterior probability that the true rate was below 5% was 98.3%.

Conclusions:

A positive artificial intelligence/sensor event from this smart eye drop bottle corresponded to objectively confirmed ocular-surface delivery in 99% of cases, with a false-assurance rate near 1%. These results validate the dose-delivery layer that motion-only adherence sensors typically lack, while the single supervised encounter establishes a controlled-condition performance ceiling and rationale for subsequent unsupervised home-use studies.