**TD-VAE**, or **Temporal Difference VAE**, is a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of [temporal difference learning](https://paperswithcode.com/method/td-lambda) used in reinforcement learning.

**Memory-Associated Differential** (**MAD**) Learning was developed to inference from the memorized facts that we already know to predict what we want to know.

Image source: [Luo et al.](https://arxiv.org/pdf/2102.05246v1.pdf)

MAD Learning

Memory-Associated Differential Learning

TD-VAE

Temporal Difference Variational Auto-Encoder

**SortCut Sinkhorn Attention** is a variant of [Sparse Sinkhorn Attention](https://paperswithcode.com/method/sparse-sinkhorn-attention) where a post-sorting truncation of the input sequence is performed, essentially performing a hard top-k operation on the input sequence blocks within the computational graph. While most attention models mainly re-weight or assign near-zero weights during training, this allows for explicitly and dynamically truncate the input sequence. Specifically:

$$ Y = \text{Softmax}\left(Q{\psi\_{S}}\left(K\right)^{T}\_{\left[:n\right]}\right)\psi\_{S}\left(V\right)\_{\left[:n\right]} $$

where $n$ is the Sortfut budget hyperparameter.

Source	Temporal Difference Variational Auto-Encoder
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com