**Sparse Switchable Normalization (SSN)** is a variant on [Switchable Normalization](https://paperswithcode.com/method/switchable-normalization) where the importance ratios are constrained to be sparse. Unlike $\ell_1$ and $\ell_0$ constraints that impose difficulties in optimization, the constrained optimization problem is turned into feed-forward computation through [SparseMax](https://paperswithcode.com/method/sparsemax), which is a sparse version of [softmax](https://paperswithcode.com/method/softmax).

**Spatially Separable Self-Attention**, or **SSSA**, is an [attention module](https://paperswithcode.com/methods/category/attention-modules) used in the [Twins-SVT](https://paperswithcode.com/method/twins-svt) architecture that aims to reduce the computational complexity of [vision transformers](https://paperswithcode.com/methods/category/vision-transformer) for dense prediction tasks (given high-resolution inputs). SSSA is composed of [locally-grouped self-attention](https://paperswithcode.com/method/locally-grouped-self-attention) (LSA) and [global sub-sampled attention](https://paperswithcode.com/method/global-sub-sampled-attention) (GSA).

Formally, spatially separable self-attention (SSSA) can be written as:

$$
\hat{\mathbf{z}}\_{i j}^{l}=\text { LSA }\left(\text { LayerNorm }\left(\mathbf{z}\_{i j}^{l-1}\right)\right)+\mathbf{z}\_{i j}^{l-1} $$

$$\mathbf{z}\_{i j}^{l}=\mathrm{FFN}\left(\operatorname{LayerNorm}\left(\hat{\mathbf{z}}\_{i j}^{l}\right)\right)+\hat{\mathbf{z}}\_{i j}^{l} $$

$$ \hat{\mathbf{z}}^{l+1}=\text { GSA }\left(\text { LayerNorm }\left(\mathbf{z}^{l}\right)\right)+\mathbf{z}^{l} $$

$$ \mathbf{z}^{l+1}=\text { FFN }\left(\text { LayerNorm }\left(\hat{\mathbf{z}}^{l+1}\right)\right)+\hat{\mathbf{z}}^{l+1}$$

$$i \in\{1,2, \ldots ., m\}, j \in\{1,2, \ldots ., n\}
$$

where LSA means locally-grouped self-attention within a sub-window; GSA is the global sub-sampled attention by interacting with the representative keys (generated by the sub-sampling functions) from each sub-window $\hat{\mathbf{z}}\_{i j} \in \mathcal{R}^{k\_{1} \times k\_{2} \times C} .$ Both LSA and GSA have multiple heads as in the standard self-attention.

Spatially Separable Self-Attention

Twins: Revisiting the Design of Spatial Attention in Vision Transformers

Sparse Switchable Normalization

SSN: Learning Sparse Switchable Normalization via SparsestMax

A **Support Vector Machine**, or **SVM**, is a non-parametric supervised learning model. For non-linear classification and regression, they utilise the kernel trick to map inputs to high-dimensional feature spaces. SVMs construct a hyper-plane or set of hyper-planes in a high or infinite dimensional space, which can be used for classification, regression or other tasks. Intuitively, a good separation is achieved by the hyper-plane that has the largest distance to the nearest training data points of any class (so-called functional margin), since in general the larger the margin the lower the generalization error of the classifier. The figure to the right shows the decision function for a linearly separable problem, with three samples on the margin boundaries, called “support vectors”. 

Source: [scikit-learn](https://scikit-learn.org/stable/modules/svm.html)

Source	SSN: Learning Sparse Switchable Normalization via SparsestMax
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com