An **Eligibility Trace** is a memory vector $\textbf{z}\_{t} \in \mathbb{R}^{d}$ that parallels the long-term weight vector $\textbf{w}\_{t} \in \mathbb{R}^{d}$. The idea is that when a component of $\textbf{w}\_{t}$ participates in producing an estimated value, the corresponding component of $\textbf{z}\_{t}$ is bumped up and then begins to fade away. Learning will then occur in that component of $\textbf{w}\_{t}$ if a nonzero TD error occurs before the trade falls back to zero. The trace-decay parameter $\lambda \in \left[0, 1\right]$ determines the rate at which the trace falls.

Intuitively, they tackle the credit assignment problem by capturing both a frequency heuristic - states that are visited more often deserve more credit - and a recency heuristic - states that are visited more recently deserve more credit.

$$E\_{0}\left(s\right) = 0 $$
$$E\_{t}\left(s\right) = \gamma\lambda{E}\_{t-1}\left(s\right) + \textbf{1}\left(S\_{t} = s\right) $$

Source: Sutton and Barto, Reinforcement Learning, 2nd Edition

Population-based intrinsically motivated goal exploration algorithms applied to real world robot learning of complex skills like tool use.

IMGEP

Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

Eligibility Trace

**Contrastive Predictive Coding v2 (CPC v2)** is a self-supervised learning approach that builds upon the original [CPC](https://paperswithcode.com/method/contrastive-predictive-coding) with several improvements. These improvements include:

- **Model capacity** - The third residual stack of [ResNet](https://paperswithcode.com/method/resnet)-101 (originally containing 23 blocks, 1024-dimensional feature maps, and 256-dimensional bottleneck layers), is converted to use 46 blocks, with 4096-dimensional feature maps and 512-dimensional bottleneck layers: ResNet-161.

- **Layer Normalization** - The authors find CPC with [batch normalization](https://paperswithcode.com/method/batch-normalization) harms downstream performance. They hypothesize this is due to batch normalization allowing large models to find a trivial solution to CPC: it introduces a dependency between patches (through the batch statistics) that can be exploited to bypass the constraints on the receptive field. They replace batch normalization with [layer normalization](https://paperswithcode.com/method/layer-normalization).

- **Predicting lengths and directions** - patches are predicted with contexts from both directions rather than just spatially underneath.

- **Patch-based Augmentation** - Utilising "color dropping" which randomly drops two of the three color channels in each patch, as well as random horizontal flips.


Consistent with prior results, this new architecture delivers better performance regardless of

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com