**VisTR** is a [Transformer](https://paperswithcode.com/method/transformer) based video instance segmentation model. It views video instance segmentation as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches.

**Holographic Reduced Representations** are a simple mechanism to represent an associative array of key-value pairs in a fixed-size vector. Each individual key-value pair is the same size as the entire associative array; the array is represented by the sum of the pairs. Concretely, consider a complex vector key $r = (a\_{r}[1]e^{iφ\_{r}[1]}, a\_{r}[2]e^{iφ\_{r}[2]}, \dots)$, which is the same size as the complex vector value x. The pair is "bound" together by element-wise complex multiplication, which multiplies the moduli and adds the phases of the elements:

$$ y = r \otimes x $$

$$ y =  \left(a\_{r}[1]a\_{x}[1]e^{i(φ\_{r}[1]+φ\_{x}[1])}, a\_{r}[2]a\_{x}[2]e^{i(φ\_{r}[2]+φ\_{x}[2])}, \dots\right) $$

Given keys $r\_{1}$, $r\_{2}$, $r\_{3}$ and input vectors $x\_{1}$, $x\_{2}$, $x\_{3}$, the associative array is:

$$c = r\_{1} \otimes x\_{1} + r\_{2} \otimes x\_{2} + r\_{3} \otimes x\_{3} $$

where we call $c$ a memory trace. Define the key inverse:

$$ r^{-1} = \left(a\_{r}[1]^{−1}e^{−iφ\_{r}[1]}, a\_{r}[2]^{−1}e^{−iφ\_{r}[2]}, \dots\right) $$

To retrieve the item associated with key $r\_{k}$, we multiply the memory trace element-wise by the vector $r^{-1}\_{k}$. For example: 

$$ r\_{2}^{−1} \otimes c = r\_{2}^{-1} \otimes \left(r\_{1} \otimes x\_{1} + r\_{2} \otimes x\_{2} + r\_{3} \otimes x\_{3}\right) $$

$$ r\_{2}^{−1} \otimes c = x\_{2} + r^{-1}\_{2} \otimes \left(r\_{1} \otimes x\_{1} + r\_{3} \otimes x3\right) $$

$$ r\_{2}^{−1} \otimes c = x\_{2} + noise $$

The product is exactly $x\_{2}$ together with a noise term. If the phases of the elements of the key vector are randomly distributed, the noise term has zero mean.

Source: [Associative LSTMs](https://arxiv.org/pdf/1602.03032.pdf)

Holographic Reduced Representation

VisTR

End-to-End Video Instance Segmentation with Transformers

**Spectral Normalization** is a normalization technique used for generative adversarial networks, used to stabilize training of the discriminator. Spectral normalization has the convenient property that the Lipschitz constant is the only hyper-parameter to be tuned.

It controls the Lipschitz constant of the discriminator $f$ by constraining the spectral norm of each layer $g : \textbf{h}\_{in} \rightarrow \textbf{h}_{out}$. The Lipschitz norm $\Vert{g}\Vert\_{\text{Lip}}$ is equal to $\sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right)$, where $\sigma\left(a\right)$ is the spectral norm of the matrix $A$ ($L\_{2}$ matrix norm of $A$):

$$ \sigma\left(a\right) = \max\_{\textbf{h}:\textbf{h}\neq{0}}\frac{\Vert{A\textbf{h}}\Vert\_{2}}{\Vert\textbf{h}\Vert\_{2}} = \max\_{\Vert\textbf{h}\Vert\_{2}\leq{1}}{\Vert{A\textbf{h}}\Vert\_{2}} $$

which is equivalent to the largest singular value of $A$. Therefore for a [linear layer](https://paperswithcode.com/method/linear-layer) $g\left(\textbf{h}\right) = W\textbf{h}$ the norm is given by $\Vert{g}\Vert\_{\text{Lip}} = \sup\_{\textbf{h}}\sigma\left(\nabla{g}\left(\textbf{h}\right)\right) = \sup\_{\textbf{h}}\sigma\left(W\right) = \sigma\left(W\right) $. Spectral normalization normalizes the spectral norm of the weight matrix $W$ so it satisfies the Lipschitz constraint $\sigma\left(W\right) = 1$:

$$ \bar{W}\_{\text{SN}}\left(W\right) = W / \sigma\left(W\right) $$

Source	End-to-End Video Instance Segmentation with Transformers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com