JMIR Publications

ABSTRACT

Background:

Triage of textual telemedical queries is a safety-critical task for medical service providers with limited remote health resources. The prioritization of patient queries containing medically severe text is necessary to optimize resource usage and provide care to those with time-sensitive needs.

Objective:

We aim to evaluate the effectiveness of transfer learning solutions on the task of telemedical triage and provide a thorough error analysis, identifying telemedicine queries which challenge state-of-the-art Natural Language Processing systems. Additionally, we aim to provide a publicly available telemedical query dataset with labels for severity classification for telemedical triage of respiratory issues.

Methods:

We annotate 573 medical queries from three online health platforms: HealthTap.com, Healthcaremagic.com and iCliniq.com. We then evaluate six transfer learning solutions utilizing various textual embedding strategies. Specifically, we first establish a baseline using a lexical classification model with Term Frequency-Inverse Document Frequency (TF-IDF) features. Next, we investigate the effectiveness of Global Vectors for Text Representation (GloVe), a pre-trained word embedding method. We evaluate the performance of GloVe embeddings in the context of Support Vector Machines, Bidirectional Long Short-Term Memory (LSTM) networks, and Hierarchical Attention Networks. Finally we evaluate the performance of contextual text embeddings using transformer-based architectures. Specifically, we evaluate Bidirectional Encoder Representations from Transformers (BERT), Bio+Clinical BERT, and Sentence-BERT (SBERT) on the telemedical triage task.

Results:

We find that a simple lexical model achieves an F1 score of 0.865 on the telemedical triage task. GloVe-based models using Support Vector Machines, Hierarchical Attention Networks , and Bidirectional LSTMs achieve a 0.8, 1.5, and 2.1-point increase in F1 score respectively. Transformer-based models such as BERT, Bio+Clinical BERT, and SBERT achieve an F1 score of 0.914, 0.904, 0.917 respectively. The highest performing model, SBERT, provides a statistically significant improvement compared to all GloVe-based and lexical baselines. However, no statistical significance is found when comparing transformer-based models. Furthermore, our error analysis reveals highly-challenging query types, including those with complex negations, temporal relationships, and patient intents. Our analysis highlights various avenues for future works which explicitly model such query challenges.

Conclusions:

We provide a new telemedical triage dataset using questions from online medical Q&A platforms. Our work shows that state-of-the-art transfer learning techniques work well on the telemedical triage task, providing significant performance increase over lexical models.