Viet-Anh on Software Logo

What is: Self-Adjusting Smooth L1 Loss?

SourceRetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Self-Adjusting Smooth L1 Loss is a loss function used in object detection that was introduced with RetinaMask. This is an improved version of Smooth L1. For Smooth L1 loss we have:

f(x)=0.5x2β if x<βf(x) = 0.5 \frac{x^{2}}{\beta} \text{ if } |x| < \beta f(x)=x0.5β otherwise f(x) = |x| -0.5\beta \text{ otherwise }

Here a point β\beta splits the positive axis range into two parts: L2L2 loss is used for targets in range [0,β][0, \beta], and L1L1 loss is used beyond β\beta to avoid over-penalizing utliers. The overall function is smooth (continuous, together with its derivative). However, the choice of control point (β\beta) is heuristic and is usually done by hyper parameter search.

Instead, with self-adjusting smooth L1 loss, inside the loss function the running mean and variance of the absolute loss are recorded. We use the running minibatch mean and variance with a momentum of 0.90.9 to update these two parameters.