A **Deep Belief Network (DBN)** is a multi-layer generative graphical model. DBNs have bi-directional connections ([RBM](https://paperswithcode.com/method/restricted-boltzmann-machine)-type connections) on the top layer while the bottom layers only have top-down connections. They are trained using layerwise pre-training. Pre-training occurs by training the network component by component bottom up: treating the first two layers as an RBM and training, then treating the second layer and third layer as another RBM and training for those parameters.

Source: [Origins of Deep Learning](https://arxiv.org/pdf/1702.07800.pdf)

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Deep_belief_network)

RESCAL model trained with a relation prediction objective on top of the 1vsAll loss

RESCAL-RP

Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations

Deep Belief Network

**Virtual Data Augmentation**, or **VDA**, is a framework for robustly fine-tuning pre-trained language model. Based on the original token embeddings, a multinomial mixture for augmenting virtual data is constructed, where a masked language model guarantees the semantic relevance and the Gaussian noise provides the augmentation diversity. Furthermore, a regularized training strategy is proposed to balance the two aspects.

Year	2009
Data Source	CC BY-SA - https://paperswithcode.com