JMIR Publications

ABSTRACT

Background:

To date, performance comparisons between men and machines have been performed in many health domains. Yet, machine learning models and human performance comparisons in audio-based respiratory diagnosis remain largely unexplored.

Objective:

The primary objective of this study is to compare human clinicians and a machine learning model in predicting COVID-19 from respiratory sound recordings.

Methods:

In this study, we compare human clinicians and a machine learning model in predicting COVID-19 from respiratory sound recordings. Prediction performance on 24 audio samples (12 tested positive) made by 36 clinicians with experience in treating COVID-19 or other respiratory illnesses is compared with predictions made by a machine learning model trained on 1,162 samples. We also investigated whether combining the predictions of the model and human experts could further enhance the performance, in terms of both accuracy and confidence.

Results:

The machine learning model outperformed the clinicians, yielding a sensitivity of 0.75 and a specificity of 0.83. Integrating clinicians’ and model’s predictions, however, could enhance performance further, achieving a sensitivity of 0.83 and a specificity of 0.92.

Conclusions:

Our findings suggest that the clinicians and the machine learning model could make better clinical decisions via a cooperative approach and achieve higher confidence in audio-based respiratory diagnosis.