Gumbel activation function, is defined using the cumulative Gumbel distribution and it can be used to perform Gumbel regression. Gumbel activation is an alternative activation function to the sigmoid or softmax activation functions and can be used to transform the unormalised output of a model to probability. Gumbel activation  $\eta_{Gumbel}$ is defined as follows:

$\eta_{Gumbel}(q_i) = exp(-exp(-q_i))$

It can be combined with Cross Entropy loss function to solve long-tailed classification problems. Gumbel Cross Entropy (GCE) is defined as follows:

$GCE(\eta_{Gumbel}(q_i),y_i) = -y_i \log(\eta_{Gumbel}(q_i))+ (1-y_i) \log(1-\eta_{Gumbel}(q_i))$

**Position-Sensitive RoIAlign** is a positive sensitive version of [RoIAlign](https://paperswithcode.com/method/roi-align) - i.e. it performs selective alignment, allowing for the learning of position-sensitive region of interest aligning.

Position-Sensitive RoIAlign

Gumbel Activation

Long-tailed Instance Segmentation using Gumbel Optimized Loss

The **Quasi-Hyperbolic Momentum Algorithm (QHM)** is a simple alteration of [momentum SGD](https://paperswithcode.com/method/sgd-with-momentum), averaging a plain [SGD](https://paperswithcode.com/method/sgd) step with a momentum step. **QHAdam** is a QH augmented version of [Adam](https://paperswithcode.com/method/adam), where we replace both of Adam's moment estimators with quasi-hyperbolic terms. QHAdam decouples the momentum term from the current gradient when updating the weights, and decouples the mean squared gradients term from the current squared gradient when updating the weights. 

In essence, it is a weighted average of the momentum and plain SGD, weighting the current gradient with an immediate discount factor $v\_{1}$ divided by a weighted average of the mean squared gradients and the current squared gradient, weighting the current squared gradient with an immediate discount factor $v\_{2}$. 

$$ \theta\_{t+1, i} = \theta\_{t, i} - \eta\left[\frac{\left(1-v\_{1}\right)\cdot{g\_{t}} + v\_{1}\cdot\hat{m}\_{t}}{\sqrt{\left(1-v\_{2}\right)g^{2}\_{t} + v\_{2}\cdot{\hat{v}\_{t}}} + \epsilon}\right], \forall{t} $$

It is recommended to set $v\_{2} = 1$ and $\beta\_{2}$ same as in Adam.

Source	Long-tailed Instance Segmentation using Gumbel Optimized Loss
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com