JMIR Publications

ABSTRACT

Background:

Sensor-based digital health technologies (sDHTs) are increasingly used to support scientific and clinical decision-making. The digital measures (DMs) they generate offer significant potential to accelerate the drug development timeline, decrease clinical trial costs, and improve access to care. However, choosing appropriate statistical methodology when conducting analytical validation (AV) of a DM is complicated, particularly for novel DMs, for which appropriate, established reference measures may not exist. More understanding of, and a standardization of approach to, AV in these scenarios is needed.

Objective:

In a prior simulation study, three statistical methods were tested for their ability to estimate a simulated relationship between a sDHT-derived DM and several clinical outcome assessment (COA) reference measures. The aim of this work was to assess the feasibility of these methods’ implementation in real data, and to examine the impact of AV study design factors on the relationships estimated.

Methods:

Four real-world datasets, captured using sDHTs, were used to prepare hypothetical AV studies that represented a range of scenarios with respect to three key study design properties: temporal coherence, construct coherence and data completeness. For each hypothetical study, two-factor, correlated-factors confirmatory factor analysis (CFA) models, and a combination of simple and multiple linear regression models, were built using the DM and reference measure data. The factor correlation, R2 and adjusted R2 statistics were calculated, in addition to the Pearson Correlation Coefficients (PCCs).

Results:

The majority of CFA models exhibited an acceptable fit according to the majority of the fit statistics employed, and each model was able to estimate a factor correlation. For each model, these correlations were greater than or equal to the corresponding PCC in magnitude. Correlations were strongest in the hypothetical studies with strong temporal and construct coherence.

Conclusions:

The performance of the measures shown in this work supports the feasibility of the selected statistical methods when implemented in real-world data. Our findings in particular support the use of CFA to assess the relationship between a novel digital measure and a COA reference measure. The observed impact of AV study design factors on the relationships estimated allowed the authors to determine practical recommendations to aid in appropriate study design for AV of novel digital measures. By utilizing a standardized methodology for evaluating novel digital measures, sDHT developers, biostatisticians and clinical researchers will be able to navigate the complex validation landscape more easily, with more certainty, and with more tools at their disposal, expediting the pathway to validation and regulatory review.