**U2-Net** is a two-level nested U-structure architecture that is designed for salient object detection (SOD).  The architecture allows the network to go deeper, attain high resolution, without significantly increasing the memory and computation cost. This is achieved by a nested U-structure: on the bottom level, with a novel ReSidual U-block (RSU) module, which is able to extract intra-stage multi-scale features without degrading the feature map resolution; on the top level, there is a [U-Net](https://paperswithcode.com/method/u-net) like structure, in which each stage is filled by a RSU block.

**FAVOR+**, or **Fast Attention Via Positive Orthogonal Random Features**, is an efficient attention mechanism used in the [Performer](https://paperswithcode.com/method/performer) architecture which leverages approaches such as kernel methods and random features approximation for approximating [softmax](https://paperswithcode.com/method/softmax) and Gaussian kernels. 

FAVOR+ works for attention blocks using matrices $\mathbf{A} \in \mathbb{R}^{L×L}$ of the form $\mathbf{A}(i, j) = K(\mathbf{q}\_{i}^{T}, \mathbf{k}\_{j}^{T})$, with $\mathbf{q}\_{i}/\mathbf{k}\_{j}$ standing for the $i^{th}/j^{th}$ query/key row-vector in $\mathbf{Q}/\mathbf{K}$ and kernel $K : \mathbb{R}^{d } × \mathbb{R}^{d} \rightarrow \mathbb{R}\_{+}$ defined for the (usually randomized) mapping: $\phi : \mathbb{R}^{d } → \mathbb{R}^{r}\_{+}$ (for some $r > 0$) as:

$$K(\mathbf{x}, \mathbf{y}) = E[\phi(\mathbf{x})^{T}\phi(\mathbf{y})] $$

We call $\phi(\mathbf{u})$ a random feature map for $\mathbf{u} \in \mathbb{R}^{d}$ . For $\mathbf{Q}^{'}, \mathbf{K}^{'} \in \mathbb{R}^{L \times r}$ with rows given as $\phi(\mathbf{q}\_{i}^{T})^{T}$ and $\phi(\mathbf{k}\_{i}^{T})^{T}$  respectively, this leads directly to the efficient attention mechanism of the form:

$$ \hat{Att\_{\leftrightarrow}}\left(\mathbf{Q}, \mathbf{K}, \mathbf{V}\right) = \hat{\mathbf{D}}^{-1}(\mathbf{Q^{'}}((\mathbf{K^{'}})^{T}\mathbf{V}))$$

where

$$\mathbf{\hat{D}} = \text{diag}(\mathbf{Q^{'}}((\mathbf{K^{'}})\mathbf{1}\_{L})) $$

The above scheme constitutes the [FA](https://paperswithcode.com/method/dfa)-part of the FAVOR+ mechanism. The other parts are achieved by:

- The R part :  The softmax kernel is approximated though trigonometric functions, in the form of a regularized softmax-kernel SMREG, that employs positive random features (PRFs).
- The OR+ part : To reduce the variance of the estimator, so we can use a smaller number of random features, different samples are entangled to be exactly orthogonal using the Gram-Schmidt orthogonalization procedure.

The details are quite technical, so it is recommended you read the paper for further information on these steps.

FAVOR+

Rethinking Attention with Performers

U2-Net

U$^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection

**Conditional DBlock** is a residual based block used in the discriminator of the [GAN-TTS](https://paperswithcode.com/method/gan-tts) architecture. They are similar to the [GBlocks](https://paperswithcode.com/method/gblock) used in the generator, but without [batch normalization](https://paperswithcode.com/method/batch-normalization). Unlike the [DBlock](https://paperswithcode.com/method/dblock), the Conditional DBlock adds the embedding of the linguistic features after the first [convolution](https://paperswithcode.com/method/convolution).

Source	U$^2$-Net: Going Deeper with Nested U-Structure for Salient Object Detection
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com