Viet-Anh on Software Logo

What is: Bootstrap Your Own Latent?

SourceBootstrap Your Own Latent - A New Approach to Self-Supervised Learning
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

BYOL (Bootstrap Your Own Latent) is a new approach to self-supervised learning. BYOL’s goal is to learn a representation yθy_θ which can then be used for downstream tasks. BYOL uses two neural networks to learn: the online and target networks. The online network is defined by a set of weights θθ and is comprised of three stages: an encoder fθf_θ, a projector gθg_θ and a predictor qθq_θ. The target network has the same architecture as the online network, but uses a different set of weights ξξ. The target network provides the regression targets to train the online network, and its parameters ξξ are an exponential moving average of the online parameters θθ.

Given the architecture diagram on the right, BYOL minimizes a similarity loss between qθ(zθ)q_θ(z_θ) and sg(zξ)sg(z'{_ξ}), where θθ are the trained weights, ξξ are an exponential moving average of θθ and sgsg means stop-gradient. At the end of training, everything but fθf_θ is discarded, and yθy_θ is used as the image representation.

Source: Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning

Image credit: Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning