**PEGASUS** proposes a transformer-based model for abstractive summarization. It uses a special self-supervised pre-training objective called gap-sentences generation (GSG) that's designed to perform well on summarization-related downstream tasks. As reported in the paper, "both GSG and MLM are applied simultaneously to this example as pre-training objectives. Originally there are three sentences. One sentence is masked with [MASK1] and used as target generation text (GSG). The other two sentences remain in the input, but some tokens are randomly masked by [MASK2]."

**Span-Based Dynamic Convolution** is a type of convolution used in the [ConvBERT](https://paperswithcode.com/method/convbert) architecture to capture local dependencies between tokens.  Kernels are generated by taking in a local span of current token, which better utilizes local dependency and discriminates different meanings of the same token (e.g., if “a” is in front of “can” in the input sentence, “can” is apparently a noun not a verb).

Specifically, with [classic convolution](https://paperswithcode.com/method/convolution), we would have fixed parameters shared for all input tokens. [Dynamic convolution](https://paperswithcode.com/method/dynamicconv) is therefore preferable because it has  higher flexibility in capturing local dependencies of different tokens. Dynamic convolution uses a kernel generator to produce different kernels for different input tokens. However, such dynamic convolution cannot differentiate the same tokens within different context and
generate the same kernels (e.g., the three “can” in Figure (b)).

Therefore the span-based dynamic convolution is developed to produce more adaptive convolution kernels by receiving an input span instead of only a single token, which enables discrimination of generated kernels for the same tokens within different context. For example, as shown in Figure (c), span-based dynamic convolution produces different kernels for different “can” tokens.

Span-Based Dynamic Convolution

ConvBERT: Improving BERT with Span-based Dynamic Convolution

PEGASUS

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

Causal inference is the process of drawing a conclusion about a causal connection based on the conditions of the occurrence of an effect. The main difference between causal inference and inference of association is that the former analyzes the response of the effect variable when the cause is changed.

Source	PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com