JMIR Publications

ABSTRACT

Epic Systems is a major provider of health information technology in the United States, providing electronic medical records (EMR) for more than 250 million patients. Epic’s platform includes predictive models for patient care, including a model that predicts a patient’s probability of being a no-show for an outpatient healthcare appointment. However, the model has not been externally validated in certain groups of patients, including people with HIV (PwH). Regular medical care for PwH is of utmost importance, missed medical appointments among PwH are independently associated with increased mortality. We conducted an external validation of Epic’s no-show model in PwH using encounter data from the University of Chicago Medicine between January 21 to March 30, 2022. We compared Epic’s predicted no-show probability at the time of the encounter to the actual outcome of these appointments. We also examined the performance of the Epic model among PwH for only HIV care appointments in the Infectious Diseases department. We further compared the no-show model among PwH for HIV care appointments to an alternate random forest model we created using a subset of 7 readily accessible features used in the Epic model and four additional features related to HIV clinical care or demographics. We identified 674 PwH who contributed 1,406 total scheduled in-person appointments during the study period. The performance of the Epic model among PwH for all appointments in any outpatient clinic had an AUC of 0.65 (0.63-0.66). When we restricted the data to include only HIV care clinic appointments, we identified 331 PwH who contributed 440 infectious disease appointments. The AUC of the Epic model in for HIV care appointments among PwH was 0.63 (0.59-0.67), there was no significant difference in performance compared to the model that included all appointments (P=0.36). The alternate model we created for PwH attending HIV care appointments had an AUC of 0.78 (0.75-0.82) a significant improvement over the Epic model restricted to HIV care appointments (P<0.001). Model performance among PwH was significantly lower than reported by Epic. We found that a model that incorporated a subset of the features used in the original Epic model along with demographic and HIV clinical information performed substantially better among PwH attending HIV care appointments. The inclusion of demographic factors seemed to improve the model performance substantially, indicating that among populations suffering from extreme disparities, such as PwH, inclusion of demographic information may be key to enhance the predicting prediction of difficulties in appointment attendance.