Viet-Anh on Software Logo

What is: Cross-View Training?

SourceSemi-Supervised Sequence Modeling with Cross-View Training
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Cross View Training, or CVT, is a semi-supervised algorithm for training distributed word representations that makes use of unlabelled and labelled examples.

CVT adds kk auxiliary prediction modules to the model, a Bi-LSTM encoder, which are used when learning on unlabeled examples. A prediction module is usually a small neural network (e.g., a hidden layer followed by a softmax layer). Each one takes as input an intermediate representation hj(xi)h^j(x_i) produced by the model (e.g., the outputs of one of the LSTMs in a Bi-LSTM model). It outputs a distribution over labels p_jθ(yx_i)p\_{j}^{\theta}\left(y\mid{x\_{i}}\right).

Each hjh^j is chosen such that it only uses a part of the input xix_i; the particular choice can depend on the task and model architecture. The auxiliary prediction modules are only used during training; the test-time prediction come from the primary prediction module that produces pθp_\theta.