Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently accepted at: JMIR AI

Date Submitted: Aug 14, 2024
Date Accepted: Nov 12, 2026

This paper has been accepted and is currently in production.

It will appear shortly on 10.2196/65341

The final accepted version (not copyedited yet) is in this tab.

Predicting adverse outcomes in COVID-19 patients: a deep learning framework using natural language processing

  • Madina Hasan; 
  • Peter Anthony Bath; 
  • Richard Pilbery; 
  • Carl Marincowitz; 
  • Suvodeep Mazumdar; 
  • Frank Hopfgartner; 
  • Richard Campbell; 
  • Tony Stone; 
  • Laura Sutton; 
  • Benjamin Thomas; 
  • Fiona Bell; 
  • Janette Turner; 
  • Katie Biggs; 
  • Jennifer Petrie; 
  • Steve Goodace

ABSTRACT

Background:

In the initial phase of the COVID-19 pandemic, patients with suspected COVID-19 required urgent and rapid assessment as rapid testing and vaccination were not available. Free-text data recorded in electronic patient records may contain important information that could help identify patients at risk of adverse outcomes.

Objective:

The aim of our study was to use free-text observations recorded by ambulance staff of patients with suspected COVID-19 to develop predictive models for patients most at risk of adverse health outcomes, i.e., dying from COVID-19 or requiring organ support.

Methods:

Natural language processing (NLP) was used to extract text-based features from emergency medical service (EMS) electronic records. Machine learning models (Support-vector machines (SVM), EXtreme Gradient Boosting (XGBoost), and a deep learning neural network, Long short-term memory (LSTM), were developed to predict patients at risk of 30-day mortality or requiring organ support; ensemble learning was used to combine the predictions from different individual models to improve performance. Model performance measures included Area Under the Receiver Operating Characteristic curve (AUC), sensitivity and specificity. Models were compared with baseline clinical accuracy, i.e., the decision made by the emergency medical staff as to whether the patient required hospital care and the predictions made by the PRIEST tool.

Results:

Overall, of the 7549 patients who were attended by an ambulance, 1330 experienced an adverse outcome (17.6%). When maximizing sensitivity, LSTM achieved the best sensitivity (0.98) and LSTM the best AUC (0.94). When maximizing specificity, the LSTM had the highest specificity (0.95) and AUC (0.94). The ensemble models with SVM and LSTM as the base learners had the best stacking results (AUC=0.97). Using the NLP tools, 1,698,484 unique words were identified in the free text from the health records. Some words e.g., covid19, cough, had similar occurrence in patients with and without adverse outcomes. Other words, e.g., good, able, were more frequent in patients who did not experience an adverse outcome and some words, e.g., unwell, sats, were more frequent in patients who experienced an adverse outcome. The words the words aids, 111 and apyrexial among the top-ranked features in both the XGB and SVM models. Among the other top-ranked words used by XGB were angina, airway, blades, and assisted. Among the other top-ranked words used by the SVM were bradycardia, chatting and apyrexial.

Conclusions:

NLP successfully extracted features from free-text observations recorded by ambulance staff about patients with suspected COVID-19 to develop models that identified patients at risk of adverse outcomes. The models performed well compared with the decision by ambulance staff and the PRIEST risk score. These methods have the potential to help identify patients at highest risk of adverse outcome during the early stages of a pandemic and can support clinical decision-making. Clinical Trial: Not applicable.


 Citation

Please cite as:

Hasan M, Bath PA, Pilbery R, Marincowitz C, Mazumdar S, Hopfgartner F, Campbell R, Stone T, Sutton L, Thomas B, Bell F, Turner J, Biggs K, Petrie J, Goodace S

Predicting adverse outcomes in COVID-19 patients: a deep learning framework using natural language processing

JMIR AI. 12/11/2026:65341 (forthcoming/in press)

DOI: 10.2196/65341

URL: https://preprints.jmir.org/preprint/65341

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.