Viet-Anh on Software Logo

What is: ProphetNet?

SourceProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

ProphetNet is a sequence-to-sequence pre-training model that introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of optimizing one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by nn-step ahead prediction that predicts the next nn tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and further help predict multiple future tokens.