Bi-attention employs the attention-in-attention (AiA) mechanism to capture second-order statistical information: the outer point-wise channel attention vectors are computed from the output of the inner channel attention.

**LOGAN** is a generative adversarial network that uses a latent optimization approach using [natural gradient descent](https://paperswithcode.com/method/natural-gradient-descent) (NGD). For the Fisher matrix in NGD, the authors use the empirical Fisher $F'$ with Tikhonov damping:

$$ F' = g \cdot g^{T} + \beta{I} $$

They also use Euclidian Norm regularization for the optimization step.

For LOGAN's base architecture, [BigGAN-deep](https://paperswithcode.com/method/biggan-deep) is used with a few modifications: increasing the size of the latent source from $186$ to $256$, to compensate the randomness of the source lost
when optimising $z$. 2, using the uniform distribution $U\left(−1, 1\right)$ instead of the standard normal distribution $N\left(0, 1\right)$ for $p\left(z\right)$ to be consistent with the clipping operation, using  leaky [ReLU](https://paperswithcode.com/method/relu) (with the slope of 0.2 for the negative part) instead of ReLU as the non-linearity for smoother gradient flow for $\frac{\delta{f}\left(z\right)}{\delta{z}}$ .

LOGAN

LOGAN: Latent Optimisation for Generative Adversarial Networks

Bi-attention

Bilinear Attention Networks for Person Retrieval

Differentiable simultaneous optimization of hyperparameters and neural network architecture. Also a [Neural Architecture Search](https://paperswithcode.com/method/neural-architecture-search) (NAS) method.

Source	Bilinear Attention Networks for Person Retrieval
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com