JMIR Publications

ABSTRACT

Background:

Women have been under-represented in clinical trials for many years. Machine learning models trained on clinical trial abstracts may capture and amplify biases in the data. Specifically, word embeddings are models that enable representing words as vectors and are the building block of most natural language processing (NLP) systems. If word embeddings are trained on clinical trial abstracts, predictive models which use the embeddings will exhibit gender performance gaps.

Objective:

To capture temporal trends in clinical trials through temporal distribution matching on contextual word embeddings (specifically, BERT) and explore its effect on the bias manifested in downstream tasks.

Methods:

We present TeDi-BERT, a method to harness the temporal trend of increasing women inclusion in clinical trials to train contextual word embeddings. We implement temporal distribution matching through an adversarial classifier, trying to distinguish old from new clinical trial abstracts based on their embeddings. The temporal distribution matching acts as a form of domain adaptation from older to more recent clinical trials. We evaluate our model on two clinical tasks: prediction of unplanned readmission to the ICU, and hospital length of stay prediction. We also conduct algorithmic analysis of the proposed method.

Results:

In readmission prediction, TeDi-BERT achieved AUC of 0.64 for female patients versus the baseline of 0.62 (P<.001), and 0.66 for male patients versus the baseline of 0.64 (P<.001). In length of stay regression, TeDi-BERT achieved a MAE of 4.56 for female patients versus 4.62 (P<.001) and 4.54 for male patients versus 4.60 (P<.001).

Conclusions:

In both clinical tasks, TeDi-BERT improved performance for female patients, as expected; but it also improved performance for male patients. Our results show that accuracy for one gender does not need to be exchanged for bias reduction, but rather that good science improves clinical results for all. Contextual word embedding models trained to capture temporal trends can help mitigate the effects of bias that changes over time in the training data.