Viet-Anh on Software Logo

What is: Normalized Temperature-scaled Cross Entropy Loss?

SourceImproved Deep Metric Learning with Multi-class N-pair Loss Objective
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

NT-Xent, or Normalized Temperature-scaled Cross Entropy Loss, is a loss function. Let sim(u,v)=uTv/uv\text{sim}\left(\mathbf{u}, \mathbf{v}\right) = \mathbf{u}^{T}\mathbf{v}/||\mathbf{u}|| ||\mathbf{v}|| denote the cosine similarity between two vectors u\mathbf{u} and v\mathbf{v}. Then the loss function for a positive pair of examples (i,j)\left(i, j\right) is :

l_i,j=logexp(sim(z_i,z_j)/τ)2N_k=11_[ki]exp(sim(z_i,z_k)/τ) \mathbb{l}\_{i,j} = -\log\frac{\exp\left(\text{sim}\left(\mathbf{z}\_{i}, \mathbf{z}\_{j}\right)/\tau\right)}{\sum^{2N}\_{k=1}\mathcal{1}\_{[k\neq{i}]}\exp\left(\text{sim}\left(\mathbf{z}\_{i}, \mathbf{z}\_{k}\right)/\tau\right)}

where 1_[ki]\mathcal{1}\_{[k\neq{i}]} \in {0,10, 1} is an indicator function evaluating to 11 iff kik\neq{i} and τ\tau denotes a temperature parameter. The final loss is computed across all positive pairs, both (i,j)\left(i, j\right) and (j,i)\left(j, i\right), in a mini-batch.

Source: SimCLR