JMIR Publications

ABSTRACT

Background:

Semantic textual similarity (STS) is a natural language processing task that involves assigning a similarity score to two snippets of text based on their meaning. This task is particularly difficult in the domain of clinical text, which often features specialised language and the frequent use of abbreviations.

Objective:

We create a natural language processing system to predict similarity scores for sentence pairs as part of the Clinical Semantic Textual Similarity track in the 2019 n2c2/OHNLP Shared Task on Challenges in Natural Language Processing for Clinical Data. We subsequently seek to analyse the intermediary token vectors extracted from our models while processing a pair of clinical sentences to identify where and how representations of semantic similarity are built in Transformer models.

Methods:

Given a clinical sentence pair, we take the average predicted similarity score across several independently fine-tuned Transformers. During our model analysis we investigate the relationship between the final model's loss and surface features of the sentence pairs and assess the decodability and representational similarity of the token vectors generated by each model.

Results:

Our model achieves a correlation of 0.87 with the ground truth similarity score, reaching 6th place out of 33 teams (with a first-place score of 0.90). In detailed qualitative and quantitative analyses of the model’s loss, we identify the system’s failure to correctly model semantic similarity when both sentence pairs contain details of medical prescriptions, as well as its general tendency to overpredict semantic similarity given significant token overlap. The token vector analysis revealed divergent representational strategies for predicting textual similarity between BERT-style models and XLNet. We also find that a large amount information relevant to predicting semantic textual similarity can be captured using a combination of a classification token and the cosine distance between sentence-pair representations in the first layer of a Transformer model that did not produce the best predictions on the test set.

Conclusions:

We design and train a system that uses state-of-the-art natural language processing models to achieve very competitive results on a new clinical semantic textual similarity dataset. As our approach uses no hand-crafted rules it serves as a strong deep learning baseline for this task. Our key contribution is a detailed analysis of the model’s outputs and an investigation of the heuristic biases learned by Transformer models. We suggest future improvements based on these findings. In our representational analysis we explore how different Transformer models converge or diverge in their representation of semantic signals as the tokens of the sentences are augmented by successive layers. This analysis sheds light on how these ‘black box’ models integrate semantic similarity information in intermediate layers, and points to new research directions in model distillation and sentence embedding extraction for applications in clinical natural language processing.