Viet-Anh on Software Logo

What is: DropConnect?

SourceRegularization of Neural Networks using DropConnect
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

DropConnect generalizes Dropout by randomly dropping the weights rather than the activations with probability 1p1-p. DropConnect is similar to Dropout as it introduces dynamic sparsity within the model, but differs in that the sparsity is on the weights WW, rather than the output vectors of a layer. In other words, the fully connected layer with DropConnect becomes a sparsely connected layer in which the connections are chosen at random during the training stage. Note that this is not equivalent to setting WW to be a fixed sparse matrix during training.

For a DropConnect layer, the output is given as:

r=a((MW)v) r = a \left(\left(M * W\right){v}\right)

Here rr is the output of a layer, vv is the input to a layer, WW are weight parameters, and MM is a binary matrix encoding the connection information where M_ijBernoulli(p)M\_{ij} \sim \text{Bernoulli}\left(p\right). Each element of the mask MM is drawn independently for each example during training, essentially instantiating a different connectivity for each example seen. Additionally, the biases are also masked out during training.