A **Cyclical Learning Rate Policy** combines a linear learning rate decay with warm restarts.

Image: [ESPNetv2](https://paperswithcode.com/method/espnetv2)

**TSRUc**, or **Transformation-based Spatial Recurrent Unit c**, is a modification of a [ConvGRU](https://paperswithcode.com/method/cgru) used in the [TriVD-GAN](https://paperswithcode.com/method/trivd-gan) architecture for video generation.

Instead of computing the reset gate $r$ and resetting $h\_{t−1}$, the TSRUc computes the parameters of a transformation $\theta$, which we use to warp $h\_{t−1}$. The rest of our model is unchanged (with $\hat{h}\_{t-1}$ playing the role of $h'\_{t}$ in $c$’s update equation from ConvGRU. The TSRUc module is described by the following equations:

$$ \theta\_{h,x} = f\left(h\_{t−1}, x\_{t}\right) $$

$$ \hat{h}\_{t-1} = w\left(h\_{t-1}; \theta\_{h, x}\right) $$

$$ c = \rho\left(W\_{c} \star\_{n}\left[\hat{h}\_{t-1};x\_{t}\right] + b\_{c} \right) $$

$$ u = \sigma\left(W\_{u} \star\_{n}\left[h\_{t-1};x\_{t}\right] + b\_{u} \right) $$

$$ h\_{t} = u \odot h\_{t-1} + \left(1-u\right) \odot c $$

In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and [ReLU](https://paperswithcode.com/method/relu) functions respectively and the $\star\_{n}$ represents a [convolution](https://paperswithcode.com/method/convolution) with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUc

Transformation-based Adversarial Video Prediction on Large-Scale Data

Cyclical Learning Rate Policy

An **XCiT Layer** is the main building block of the [XCiT](https://paperswithcode.com/method/xcit) architecture which uses a [cross-covariance attention]() operator as its principal operation. The XCiT layer consists of three main blocks, each preceded by [LayerNorm](https://paperswithcode.com/method/layer-normalization) and followed by a [residual connection](https://paperswithcode.com/method/residual-connection): (i) the core [cross-covariance attention](https://paperswithcode.com/method/cross-covariance-attention) (XCA) operation, (ii) the [local patch interaction](https://paperswithcode.com/method/local-patch-interaction) (LPI) module, and (iii) a [feed-forward network](https://paperswithcode.com/method/feedforward-network) (FFN). By transposing the query-key interaction, the computational complexity of XCA is linear in the number of data elements N, rather than quadratic as in conventional self-attention.

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com