Viet-Anh on Software Logo

What is: DropPath?

SourceFractalNet: Ultra-Deep Neural Networks without Residuals
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Just as dropout prevents co-adaptation of activations, DropPath prevents co-adaptation of parallel paths in networks such as FractalNets by randomly dropping operands of the join layers. This discourages the network from using one input path as an anchor and another as a corrective term (a configuration that, if not prevented, is prone to overfitting). Two sampling strategies are:

  • Local: a join drops each input with fixed probability, but we make sure at least one survives.
  • Global: a single path is selected for the entire network. We restrict this path to be a single column, thereby promoting individual columns as independently strong predictors.