Varifocal Loss: Learning IoU-Aware Classification Scores for Detection

Varifocal Loss (VFL) trains a dense object detector to predict an IoU-aware classification score: a score that represents both the presence of an object and the localization quality of its predicted box.

For the target class, the training target $q$ is the IoU between the predicted box and its matched ground-truth box. For background and non-target classes, $q=0$. If $p$ is the predicted class probability, the paper defines

Why it differs from focal loss

Focal loss was designed to reduce the influence of numerous easy negatives. Varifocal Loss keeps that behavior for negatives through $p^\gamma$, but treats positives asymmetrically:

• Positive targets are continuous IoU values rather than one-hot labels.
• High-quality positives receive more weight through the outer factor $q$.
• Positive examples are not down-weighted with the focal factor used for negatives.

This trains the classification score itself to rank well-localized boxes above poorly localized ones. That alignment matters because detectors select top candidates and apply non-maximum suppression using their scores.

Numerical and training details

The paper uses $\alpha=0.75$ and $\gamma=2.0$ in its main experiments. Those values belong to its detector and training recipe rather than to the mathematical definition, so a different detector or assignment strategy may need different settings.

Scope

Varifocal Loss was proposed as one part of VarifocalNet, alongside a star-shaped feature representation and box-refinement branch. Do not attribute the detector's complete reported improvement to the loss alone: the paper's ablation separates these components.

Primary source

• Haoyang Zhang et al., VarifocalNet: An IoU-Aware Dense Object Detector, CVPR 2021.