Viet-Anh on Software Logo

What is: Parameterized ReLU?

SourceDelving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

A Parametric Rectified Linear Unit, or PReLU, is an activation function that generalizes the traditional rectified unit with a slope for negative values. Formally:

f(y_i)=y_i if y_i0f\left(y\_{i}\right) = y\_{i} \text{ if } y\_{i} \ge 0 f(y_i)=a_iy_i if y_i0f\left(y\_{i}\right) = a\_{i}y\_{i} \text{ if } y\_{i} \leq 0

The intuition is that different layers may require different types of nonlinearity. Indeed the authors find in experiments with convolutional neural networks that PReLus for the initial layer have more positive slopes, i.e. closer to linear. Since the filters of the first layers are Gabor-like filters such as edge or texture detectors, this shows a circumstance where positive and negative responses of filters are respected. In contrast the authors find deeper layers have smaller coefficients, suggesting the model becomes more discriminative at later layers (while it wants to retain more information at earlier layers).