SAGA is a method in the spirit of SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient algorithms with fast linear convergence rates. SAGA improves on the theory behind SAG and SVRG, with better theoretical convergence rates, and has support for composite objectives where a proximal operator is used on the regulariser. Unlike SDCA, SAGA supports non-strongly convex problems directly, and is adaptive to any inherent strong convexity of the problem.

**Zoneout** is a  method for regularizing [RNNs](https://paperswithcode.com/methods/category/recurrent-neural-networks). At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like [dropout](https://paperswithcode.com/method/dropout), zoneout uses random noise to train a pseudo-ensemble, improving generalization.
But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward [stochastic depth](https://paperswithcode.com/method/stochastic-depth) networks.

Zoneout

Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations

SAGA

SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

In relation-aware global attention (RGA) stresses the importance of global structural information provided by pairwise relations, and uses it to produce attention maps. 

RGA comes in two forms,  spatial RGA (RGA-S) and channel RGA (RGA-C). RGA-S first reshapes the input feature map $X$ to $C\times (H\times W)$ and the pairwise relation matrix $R \in \mathbb{R}^{(H\times W)\times (H\times W)}$ is computed using 
\begin{align}
    Q &= \delta(W^QX) 
\end{align}
\begin{align}
    K &= \delta(W^KX) 
\end{align}
\begin{align}
    R &= Q^TK
\end{align}
The relation vector $r_i$ at position $i$ is defined by stacking  pairwise relations at all positions:
\begin{align}
    r_i = [R(i, :); R(:,i)]    
\end{align}
and the spatial relation-aware feature $y_i$ can be written as
\begin{align}
    Y_i = [g^c_\text{avg}(\delta(W^\varphi x_i)); \delta(W^\phi r_i)]
\end{align}
where $g^c_\text{avg}$ denotes global average pooling in the channel domain. Finally, the spatial attention score at position $i$ is given by 
\begin{align}
    a_i = \sigma(W_2\delta(W_1y_i))
\end{align}
RGA-C has the same form as RGA-S, except for taking the input feature map as a set of $H\times W$-dimensional features.

RGA uses global relations to generate the attention score for each feature node,  so provides valuable structural information and significantly enhances the representational power. RGA-S and RGA-C are flexible enough to be used in any CNN network; Zhang et al. propose  using them jointly in sequence to better capture both spatial and cross-channel relationships.

Source	SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com