Accepted for/Published in: JMIR Medical Informatics
Date Submitted: Jul 7, 2023
Date Accepted: May 4, 2024
Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice
ABSTRACT
Background:
Integrating machine learning models into clinical practice presents a challenge of maintaining their efficacy over time. While the existing literature offers valuable strategies for detecting declining model performance, there is a need to document the broader challenges and solutions associated with real-world development and integration of model monitoring solutions.
Objective:
This work details the development and utilization of a platform for monitoring the performance of a production-level machine learning model operating in Mayo Clinic. The aim of this work is to provide a series of considerations and guidelines necessary for integrating such a platform into a team’s technical infrastructure and workflow. We document our experiences with this integration process and discuss the broader challenges encountered with real-world implementation and maintenance. Source code for the platform is also included.
Methods:
Our monitoring platform was built as an R shiny application; developed and implemented over the course of 6 months. The platform has been utilized and maintained for 2 years and is still in use as of July 2023.
Results:
The considerations necessary for the implementation of the monitoring platform center around four pillars: Feasibility – what resources can be utilized for platform development?; Design – through what statistics or models will the model be monitored, and how will these results be efficiently displayed to the end-user?; Implementation – how will this platform be built and where will it exist within the IT ecosystem?; and Policy – based on monitoring feedback, when and what actions will be taken to fix problems and how will these problems be translated to clinical staff?
Conclusions:
While much of the literature surrounding machine learning performance monitoring emphasizes methodological approaches for capturing changes in performance, there remain a battery of other challenges and considerations that must be addressed for successful real-world implementation.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.