Viet-Anh on Software Logo

What is: Fraternal Dropout?

SourceFraternal Dropout
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Fraternal Dropout is a regularization method for recurrent neural networks that trains two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. This encourages the representations of RNNs to be invariant to dropout mask, thus being robust.