Some object detection datasets contain an overwhelming number of easy examples and a small number of hard examples. Automatic selection of these hard examples can make training more
effective and efficient. **OHEM**, or **Online Hard Example Mining**, is a bootstrapping technique that modifies [SGD](https://paperswithcode.com/method/sgd) to sample from examples in a non-uniform way depending on the current loss of each example under consideration. The method takes advantage of detection-specific problem structure in which each SGD mini-batch consists of only one or two images, but thousands of candidate examples. The candidate examples are subsampled according to a distribution
that favors diverse, high loss instances.

BYOL (Bootstrap Your Own Latent) is a new approach to self-supervised learning. BYOL’s goal is to learn a representation $y_θ$ which can then be used for downstream tasks. BYOL uses two neural networks to learn: the online and target networks. The online network is defined by a set of weights $θ$ and is comprised of three stages: an encoder $f_θ$, a projector $g_θ$ and a predictor $q_θ$. The target network has the same architecture
as the online network, but uses a different set of weights $ξ$. The target network provides the regression
targets to train the online network, and its parameters $ξ$ are an exponential moving average of the
online parameters $θ$.

Given the architecture diagram on the right, BYOL minimizes a similarity loss between $q_θ(z_θ)$ and $sg(z'{_ξ})$, where $θ$ are the trained weights, $ξ$ are an exponential moving average of $θ$ and $sg$ means stop-gradient. At the end of training, everything but $f_θ$ is discarded, and $y_θ$ is used as the image representation.

Source: [Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning](https://paperswithcode.com/paper/bootstrap-your-own-latent-a-new-approach-to-1)

Image credit: [Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning](https://paperswithcode.com/paper/bootstrap-your-own-latent-a-new-approach-to-1)

BYOL

Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning

OHEM

Training Region-based Object Detectors with Online Hard Example Mining

Spatial Broadcast Decoder is an architecture that aims to improve disentangling, reconstruction accuracy, and generalization to held-out regions in data space. It provides a particularly dramatic
benefit when applied to datasets with small objects.

Source: [Watters et al.](https://arxiv.org/pdf/1901.07017v2.pdf)

Image source: [Watters et al.](https://arxiv.org/pdf/1901.07017v2.pdf)

Source	Training Region-based Object Detectors with Online Hard Example Mining
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com