**Contextual Decomposition Explanation Penalization (CDEP)** is a method which leverages existing explanation techniques for neural networks in order to prevent a model from learning
unwanted relationships and ultimately improve predictive accuracy. Given particular importance
scores, CDEP works by allowing the user to directly penalize importances of certain features, or
interactions. This forces the neural network to not only produce the correct prediction, but also the
correct explanation for that prediction

**Slime Mould Algorithm** (**SMA**) is a new stochastic optimizer proposed based on the oscillation mode of slime mould in nature. SMA has several new features with a unique mathematical model that uses adaptive weights to simulate the process of producing positive and negative feedback of the propagation wave of slime mould based on bio-oscillator to form the optimal path for connecting food with excellent exploratory ability and exploitation propensity.

🔗 The source codes of SMA are publicly available at [https://aliasgharheidari.com/SMA.html](https://aliasgharheidari.com/SMA.html)

CDEP

Interpretations are useful: penalizing explanations to align neural networks with prior knowledge

Introduced by Hinton et al. in 2012, dropout has stood the test of time as a regularizer for preventing overfitting in neural networks. In this study, we demonstrate that dropout can also mitigate underfitting when used at the start of training. During the early phase, we find dropout reduces the directional variance of gradients across mini-batches and helps align the mini-batch gradients with the entire dataset's gradient. This helps counteract the stochasticity of SGD and limit the influence of individual batches on model training. Our findings lead us to a solution for improving performance in underfitting models - early dropout: dropout is applied only during the initial phases of training, and turned off afterwards. Models equipped with early dropout achieve lower final training loss compared to their counterparts without dropout. Additionally, we explore a symmetric technique for regularizing overfitting models - late dropout, where dropout is not used in the early iterations and is only activated later in training. Experiments on ImageNet and various vision tasks demonstrate that our methods consistently improve generalization accuracy. Our results encourage more research on understanding regularization in deep learning and our methods can be useful tools for future neural network training, especially in the era of large data. Code is available at https://github.com/facebookresearch/dropout .

Source	Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com