Options
The Effect of Pretraining on Extractive Summarization for Scientific Documents
Journal
2nd Workshop on Scholarly Document Processing, SDP 2021 - Proceedings of the Workshop
Date Issued
2021-01-01
Author(s)
Gupta, Yash
Ammanamanchi, Pawan Sasanka
Bordia, Shikha
Manoharan, Arjun
Mittal, Deepak
Pasunuru, Ramakanth
Shrivastava, Manish
Singh, Maneesh
Bansal, Mohit
Jyothi, Preethi
Abstract
Large pretrained models have seen enormous success in extractive summarization tasks. We investigate, here, the influence of pretraining on a BERT-based extractive summarization system for scientific documents. We derive performance improvements using an intermediate pretraining step that leverages existing summarization datasets and report state-of-theart results on a recently released scientific summarization dataset, SCITLDR. We systematically analyze the intermediate pretraining step by varying the size and domain of the pretraining corpus, changing the length of the input sequence in the target task and varying target tasks. We also investigate how intermediate pretraining interacts with contextualized word embeddings trained on different domains.