JMIR Publications

ABSTRACT

Background:

Diagnostic errors pose significant health risks and contribute to patient mortality. With the growing accessibility of electronic health records, machine learning models offer a promising avenue for enhancing diagnosis quality. Current research has primarily focused on a limited set of diseases with ample training data, neglecting diagnostic scenarios with limited data availability.

Objective:

This study aims to develop an information retrieval (IR) based framework that accommodates data sparsity to facilitate broader diagnostic decision support.

Methods:

We present an IR-based diagnostic decision support framework called CliniqIR. It employs clinical text records, the Unified Medical Language System (UMLS) Metathesaurus, and 33M PubMed abstracts to classify a broad spectrum of diagnoses independent of training data availability. We compare CliniqIR's performance to pre-trained clinical transformer models (like ClinicalBERT) in supervised and zero-shot settings. Subsequently, we combine the strength of supervised fine-tuned ClinicalBERT and CliniqIR to build an ensemble framework that delivers state-of-the-art diagnostic predictions.

Results:

CliniqIR returns the correct diagnosis for a DC3 case among its top-3 predictions, on average, on a rare disease dataset (DC3) with no training data. On the MIMIC-III dataset, CliniqIR outperforms ClinicalBERT in predicting diagnoses with fewer than five training samples by an average Mean Reciprocal Rank (MRR) of 9%. In a zero-shot setting, where no specific training was conducted, CliniqIR also outperforms the pre-trained transformer models by an MRR of 10%. Furthermore, our ensemble framework surpassed the individual constituent models by a minimum of 8% in MRR.

Conclusions:

Our experiments highlight the importance of IR in leveraging unstructured knowledge resources to identify infrequently encountered diagnoses. In addition, our ensemble framework benefits from combining the complementary strengths of the supervised and retrieval-based models to diagnose a broad spectrum of diseases.