**Virtual Data Augmentation**, or **VDA**, is a framework for robustly fine-tuning pre-trained language model. Based on the original token embeddings, a multinomial mixture for augmenting virtual data is constructed, where a masked language model guarantees the semantic relevance and the Gaussian noise provides the augmentation diversity. Furthermore, a regularized training strategy is proposed to balance the two aspects.

A **Deep Belief Network (DBN)** is a multi-layer generative graphical model. DBNs have bi-directional connections ([RBM](https://paperswithcode.com/method/restricted-boltzmann-machine)-type connections) on the top layer while the bottom layers only have top-down connections. They are trained using layerwise pre-training. Pre-training occurs by training the network component by component bottom up: treating the first two layers as an RBM and training, then treating the second layer and third layer as another RBM and training for those parameters.

Source: [Origins of Deep Learning](https://arxiv.org/pdf/1702.07800.pdf)

Image Source: [Wikipedia](https://en.wikipedia.org/wiki/Deep_belief_network)

Deep Belief Network

Virtual Data Augmentation

Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models

Simulations of multi-modal distributions can be very costly and often lead to unreliable predictions. To accelerate the computations, we propose to sample from a flattened distribution to accelerate the computations and estimate the importance weights between the original distribution and the flattened distribution to ensure the correctness of the distribution.

Source	Virtual Data Augmentation: A Robust and General Framework for Fine-tuning Pre-trained Models
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com