JMIR Publications

ABSTRACT

Background:

With the increasing availability of data, computing resources, and easier-to-use software libraries, artificial intelligence (AI) is increasingly utilized in disease detection and prediction, including Parkinson’s Disease (PD). Despite the large number of studies published every year, very few AI systems are adopted for patient use. This review highlights the tools and techniques to minimize the gap between reported performance in research papers and the real-life applicability of machine learning models in detecting and predicting diseases such as PD.

Objective:

This review focuses on standard machine learning model reporting practices in PD prediction. Previous studies acknowledge that lack of standardization reporting and insufficient information create barriers for the research community. We synthesized the information of 113 studies to identify recurring issues or areas where reporting practice falls short. This identification allows the scientific community to recognize the specific pitfalls and can guide future efforts to enhance the reporting standard. We emphasize the importance of adhering to these practices to foster research progress and ensure the credibility use of machine learning models in real-world applications.

Methods:

We conducted a systematic review of studies in 2020 and 2021 that use machine learning models to diagnose PD or to track PD progression. Using a predefined standard query and appropriate exclusion criteria, we searched PUBMED and found 113 papers that specifically used machine learning for the classification or regression prediction of PD or PD-related symptoms.

Results:

Notable limitations include observing that only 27.4% of papers used a hold-out test set to avoid potentially inflated accuracies, and approximately half of the papers without a hold-out test set did not state this as a potential concern. Surprisingly, 38.9% of papers did not report on how or if models were tuned, and an additional 27.4% used ad hoc model tuning, which is generally frowned upon in machine learning model optimization. Only 11.5% of papers performed direct comparisons of results with other models, severely limiting the interpretation of results.

Conclusions:

These data-driven detection and prediction models perform well in publication but fail to perform in practice, which can imply modern AI methods are limited. However, this lack of external validity is primarily the result of poor practices in using machine learning, which hinders clinical adoption of AI, even for challenges that may ultimately require such data-driven machine learning approaches.