JMIR Publications

ABSTRACT

Background:

There is increasing attention for machine learning based clinical decision support systems (CDSS), but their added value and pitfalls are very rarely evaluated at the frontline of clinical practice. We have implemented a CDSS to aid general practitioners (GP) with the treatment of patients with urinary tract infections (UTI). UTIs are a large health burden worldwide and the scientific evidence for clinically effective treatments with increased risk of a complicated UTI is limited.

Objective:

In this study, we prospectively assess the impact of this CDSS on treatment success and change in antibiotic prescription behavior of the physician. Doing so, we hope to identify drivers and obstacles for positively impacting the quality of healthcare practice with machine learning.

Methods:

The CDSS was developed as a joint effort by Pacmed, Nivel and LUMC. The CDSS presents the expected outcomes of treatments together with context information needed to assess the expected outcomes well. Treatment success was defined as a subsequent period of 28 days where no new antibiotic treatment for the UTI was needed. In this prospective observational study, 36 primary care practices used the software for a period of four months, starting in November 2017. Twenty-nine control practices were identified through a propensity score matching procedure. All analyses have been done on electronic health records from the Nivel Primary Care database. Patients for which the software has been used have been identified in the Nivel database through a sequential matching procedure using the CDSS usage data. To evaluate treatment success, we have compared the proportion of successful treatments prior and during the study within the treatment arm. The same analysis has been done for the control practices and for the subgroup of patients we were sure of the software has been used for. All analyses were statistically tested by two-sided z-tests with an alpha level of .05. In assessing the difference of treatment success for several patient subgroups, Bonferroni corrections were applied. Lastly, the antibiotic prescription behavior of the physicians was analyzed through the same z-tests.

Results:

In the treatment practices, 4998 patients were included in the period of time before the implementation study, 3422 patients were included during the implementation period. In the control practices, 5044 patients were included before the implementation period, 3360 patients were included during. The proportion of successful treatments increased significantly from 75% to 80% on average in the treatment practices (z=5.47, p<.001). In the control practices, no significant difference was detected (76% before and 76% during the pilot, z=0.02, p=0.98). We have been able to identify 734 out of 1200 patients in the CDSS usage database in the Nivel database. For these patients, of whom we are certain the software has been used for, the proportion of successful treatments during the study was 83%. This is a statistically significant difference with the 75% of successful treatments prior to the study in the treatment practices (z=4.95, p<.001).

Conclusions:

The introduction of the CDSS as intervention in the 36 treatment practices was associated with statistically significant improved treatment success. We have excluded temporal effects and validated the result with the subgroup analysis in patients for whom we are certain the software was used. The study shows important strengths and points of attention for the development and implementation of a machine learning based CDSS in clinical practice. Clinical Trial: The trial was registered on ClinicalTrials.gov under the Identifier NCT04408976.