**Conditional Instance Normalization** is a normalization technique where all convolutional weights of a style transfer network are shared across many styles.  The goal of the procedure is transform
a layer’s activations $x$ into a normalized activation $z$ specific to painting style $s$. Building off
[instance normalization](https://paperswithcode.com/method/instance-normalization), we augment the $\gamma$ and $\beta$ parameters so that they’re $N \times C$ matrices, where $N$ is the number of styles being modeled and $C$ is the number of output feature maps. Conditioning on a style is achieved as follows:

$$ z = \gamma\_{s}\left(\frac{x - \mu}{\sigma}\right) + \beta\_{s}$$

where $\mu$ and $\sigma$ are $x$’s mean and standard deviation taken across spatial axes and $\gamma\_{s}$ and $\beta\_{s}$ are obtained by selecting the row corresponding to $s$ in the $\gamma$ and $\beta$ matrices. One added benefit of this approach is that one can stylize a single image into $N$ painting styles with a single feed forward pass of the network with a batch size of $N$.

**Factorized Dense Synthesized Attention** is a synthesized attention mechanism, similar to [dense synthesized attention](https://paperswithcode.com/method/dense-synthesized-attention), but we factorize the outputs to reduce parameters and prevent overfitting. It was proposed as part of the [Synthesizer](https://paperswithcode.com/method/synthesizer) architecture. The factorized variant of the dense synthesizer can be expressed as follows:

$$A, B = F\_{A}\left(X\_{i}\right), F\_{B}\left(X\_{i}\right)$$

where $F\_{A}\left(.\right)$ projects input $X\_{i}$ into $a$ dimensions, $F\_B\left(.\right)$ projects $X\_{i}$ to $b$ dimensions, and $a \text{ x } b = l$. The output of the factorized module is now written as:

$$ Y = \text{Softmax}\left(C\right)G\left(X\right) $$

where $C = H\_{A}\left(A\right) * H\_{B}\left(B\right)$, where $H\_{A}$, $H\_{B}$ are tiling functions and $C \in \mathbb{R}^{l \text{ x } l}$. The tiling function simply duplicates the vector $k$ times, i.e., $\mathbb{R}^{l} \rightarrow \mathbb{R}^{lk}$. In this case, $H\_{A}\left(\right)$ is a projection of $\mathbb{R}^{a} \rightarrow \mathbb{R}^{ab}$ and $H\_{B}\left(\right)$ is a projection of $\mathbb{R}^{b} \rightarrow \mathbb{R}^{ba}$. To avoid having similar values within the same block, we compose the outputs of $H\_{A}$ and $H\_{B}$.

Factorized Dense Synthesized Attention

Synthesizer: Rethinking Self-Attention in Transformer Models

Conditional Instance Normalization

A Learned Representation For Artistic Style

A **Deactivable Skip Connection** is a type of skip connection which, instead of concatenating the encoder features
(red) and decoder features (blue), as with [standard skip connections](https://paperswithcode.com/methods/category/skip-connections), it instead fuses the encoder features with part of the decoder features (light blue), to be able to deactivate this operation when needed.

Source	A Learned Representation For Artistic Style
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com