JMIR Publications

ABSTRACT

Background:

Machine reading comprehension (MRC) tasks have been developed, with state-of-the-art extractive methods having achieved comparable or better accuracy than human performance on benchmark datasets. However, such models are not as successful when adapted to complex domains such as healthcare. One of the main reasons is that the context that the MRC model needs to process when operating in a complex domain can be much larger compared to an average open domain context. A potential solution to this problem is reducing the input context to the MRC model by extracting only the necessary parts from the original context.

Objective:

This study aims to develop a method for extracting useful contexts from long articles for Question Answering to enable the MRC model to work more efficiently and accurately.

Methods:

Existing approaches to context extraction in MRC were based on sentence selection strategies, in which the models were trained to find the sentence(s) containing the answer. We found that using just the sentences containing the answer was insufficient for the MRC model to predict correctly. We conducted a series of empirical studies and observed a strong relation between the usefulness of the context and the confidence score output by the MRC model. We also found a strong relationship between the model’s confidence in its predictions and the prediction accuracy. We propose a method to estimate the utility of each sentence in a context in answering the question, then extract a new, shorter context according to these estimations. We generated a dataset to train two models for estimating sentence utility, based on which we select more precise contexts that improve the MRC model’s performance.

Results:

We demonstrated our approach on the COVID-QA and BioASQ datasets and showed that our proposed context extraction approach benefits the downstream MRC model. First, our proposed method significantly reduces the inference time of the entire QA system by 6 - 7 times. Second, our approach helps the MRC model predict the answer more accurately compared to using the original context (2.76% increase in F1 score for COVID-QA and 8.14% increase in F1 score for BioASQ). We also found a potential problem in extractive Transformers MRC models where they predict poorly when given a shorter, more precise context in some cases.

Conclusions:

Utilization of the proposed context extraction method allows the MRC model to achieve improved prediction accuracy and significantly reduces MRC inference time.