JMIR Publications

ABSTRACT

Background:

Machine learning has recently helped create models for predicting medical events, mostly using private datasets. The MIMIC-3 dataset, which is public and contains detailed data on over 40,000 intensive care unit patients, stands out as it can help develop better models using not only structured data but also text from medical records.

Objective:

This study aimed to build and test a machine learning model using the MIMIC-3 dataset to determine the effectiveness of information extracted from electronic medical record text using named entity recognition (NER), specifically QuickUMLS, for predicting important medical events. Using the prediction of extended-spectrum beta-lactamase (ESBL)-producing bacterial infections as an example, this study shows how open data sources and simple technology can be useful for making clinically meaningful predictions.

Methods:

The MIMIC-3 dataset, which offers a wealth of information, including demographics, vital signs, laboratory results, and textual data, such as discharge summaries, was used. This study specifically targeted patients diagnosed with Klebsiella pneumoniae or Escherichia coli infection. Predictions were based on ESBL-producing bacterial standards and the minimum inhibitory concentration criteria. Both the structured data and extracted patient histories were used as predictors. Two models, an L1-regularized logistic regression model and a LightGBM model, were evaluated using the receiver operating characteristic area under the curve (ROC-AUC) and the precision-recall curve area under the curve (PR-AUC).

Results:

Of 46,520 MIMIC-3 patients, 4,046 were identified using bacterial culture tests, indicating the presence of K. pneumoniae or E. coli. After excluding patients who lacked discharge summary text, 3,614 patients remained. The L1-penalized model, with variables from only the structured data, displayed an ROC-AUC of 0.646 and a PR-AUC of 0.307. The LightGBM model with variables from the structured and text data outperformed the former, with an ROC-AUC of 0.707 and PR-AUC of 0.369. Key contributors to the LightGBM model included patient age, duration since hospital admission, and specific medical history such as diabetes.

Conclusions:

The structured data-based model showed improved performance compared to the reference models. Performance was further improved when textual medical history was included. Compared to other models predicting drug-resistant bacteria, the results of this study ranked in the middle. Some misidentifications, potentially due to the limitations of QuickUMLS, may have affected the accuracy of the mode. This study successfully developed a predictive model for ESBL-producing bacterial infections using the MIMIC-3 dataset, yielding results consistent with existing literature. This model stands out for its transparency and reliance on open data and open NER technology. The performance of the model was enhanced using textual information. With advancements in natural language processing tools, such as BERT and GPT, the extraction of medical data from textual sources holds substantial potential for future model optimization.