JMIR Publications

ABSTRACT

Background:

Question-driven summarization has become a practical and accurate approach to summarizing the source document. The generated summary should be concise and consistent with the concerned question and thus it could be regarded as the answer to the non-factoid question. Existing methods do not fully exploit question information over documents and dependencies across sentences. Besides, most existing summarization evaluation tools like ROUGE calculate N-gram overlaps between the generated summary and the reference summary, while neglecting the factual consistency problem.

Objective:

This proposes a novel question-driven abstractive summarization model based on Transformer, including a two-step attention mechanism and an overall integration mechanism, which can generate concise and consistent summaries for non-factoid QA.

Methods:

Specifically, the two-step attention mechanism is proposed to exploit the mutual information both of question to context and sentence over other sentences. We further introduce an overall integration mechanism and a novel pointer network for information integration. We conduct a question answering task to evaluate the factual consistency between the generated summary and the reference summary.

Results:

The experimental results of question-driven summarization on PubMedQA dataset show that our model achieves a ROUGE-1, ROUGE-2 and ROUGE-L measure of 36.01, 15.59 and 30.22, which is superior to the state-of-the-art methods with a gain of 0.79 (absolute) in ROUGE-2 score. The question answering task demonstrates that the generated summaries of our model have better factual constancy. Our method achieves 94.20% accuracy and 77.57% F1 score.

Conclusions:

Our proposed question-driven summarization model effectively exploits the mutual information among the question, document and summary to generate concise and consistent summaries.