JMIR Publications

ABSTRACT

Background:

The COVID-19 pandemic has highlighted the need for robust and adaptable diagnostic tools capable of detecting the disease from diverse and continuously evolving data sources. Machine learning models, particularly convolutional neural networks (CNNs), have shown promise in this regard. However, the dynamic nature of real-world data can lead to model drift, where the model's performance degrades over time as the underlying data distribution changes. Addressing this challenge is crucial to maintaining the accuracy and reliability of these models in ongoing diagnostic applications.

Objective:

This study aims to develop a comprehensive framework that not only monitors model drift over time but also employs adaptation mechanisms to mitigate performance fluctuations in COVID-19 detection models trained on dynamic audio data.

Methods:

Two crowd-sourced COVID-19 audio datasets, the COVID-19 Sounds and COSWARA datasets, were used for development and evaluation purposes. Each dataset was divided into two distinct periods: development and post-development. A baseline CNN model was initially trained and evaluated using data (i.e., cough recordings) from the development period. To detect changes in data distributions and the model’s performance between these periods, the maximum mean discrepancy (MMD) distance was employed. Upon detecting significant drift, a retraining procedure was triggered to update the baseline model. The study explored two model adaptation approaches: unsupervised domain adaptation (UDA) and active learning (AL), both of which were comparatively assessed.

Results:

The application of the UDA approach led to performance improvement in terms of the balanced accuracy by up to 22% and 24% for the COVID-19 Sounds and COSWARA datasets, respectively. The AL approach yielded even greater improvement, corresponding to a balanced accuracy increase of up to 30% and 60% for the two datasets, respectively.

Conclusions:

The proposed framework successfully addresses the challenge of model drift in COVID-19 detection by enabling continuous adaptation to evolving data distributions. This approach ensures sustained model performance over time, contributing to the development of robust and adaptable diagnostic tools for COVID-19 and potentially other infectious diseases.