Population-based intrinsically motivated goal exploration algorithms applied to real world robot learning of complex skills like tool use.

The **Deep LSTM Reader** is a neural network for reading comprehension. We feed documents one word at a time into a Deep [LSTM](https://paperswithcode.com/method/lstm) encoder, after a delimiter we then also feed the query into the encoder. The model therefore processes each document query pair as a single long sequence. Given the embedded document and query the network predicts which token in the document answers the query.

The model consists of a Deep LSTM cell with skip connections from each input $x\left(t\right)$ to every hidden layer, and from every hidden layer to the output $y\left(t\right)$:

$$x'\left(t, k\right) = x\left(t\right)||y'\left(t, k - 1\right) \text{,  } y\left(t\right) = y'\left(t, 1\right)|| \dots ||y'\left(t, K\right) $$

$$ i\left(t, k\right) =  \left(W\_{kxi}x'\left(t, k\right) + W\_{khi}h(t - 1, k) + W\_{kci}c\left(t - 1, k\right) + b\_{ki}\right) $$

$$ f\left(t, k\right) =  \left(W\_{kxf}x\left(t\right) + W\_{khf}h\left(t - 1, k\right) + W\_{kcf}c\left(t - 1, k\right) + b\_{kf}\right) $$

$$ c\left(t, k\right) = f\left(t, k\right)c\left(t - 1, k\right) + i\left(t, k\right)\text{tanh}\left(W\_{kxc}x'\left(t, k\right) + W\_{khc}h\left(t -  1, k\right) + b\_{kc}\right) $$

$$ o\left(t, k\right) =  \left(W\_{kxo}x'\left(t, k\right) + W\_{kho}h\left(t - 1, k\right) + W\_{kco}c\left(t, k\right) + b\_{ko}\right) $$

$$ h\left(t, k\right) = o\left(t, k\right)\text{tanh}\left(c\left(t, k\right)\right) $$

$$ y'\left(t, k\right) = W\_{kyh}\left(t, k\right) + b\_{ky} $$

where || indicates vector concatenation, $h\left(t, k\right)$ is the hidden state for layer $k$ at time $t$, and $i$, $f$, $o$ are the input, forget, and output gates respectively. Thus our Deep LSTM Reader is defined by $g^{\text{LSTM}}\left(d, q\right) = y\left(|d|+|q|\right)$ with input $x\left(t\right)$ the concatenation of $d$ and $q$ separated by the delimiter |||.

Deep LSTM Reader

Teaching Machines to Read and Comprehend

IMGEP

Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

An **Eligibility Trace** is a memory vector $\textbf{z}\_{t} \in \mathbb{R}^{d}$ that parallels the long-term weight vector $\textbf{w}\_{t} \in \mathbb{R}^{d}$. The idea is that when a component of $\textbf{w}\_{t}$ participates in producing an estimated value, the corresponding component of $\textbf{z}\_{t}$ is bumped up and then begins to fade away. Learning will then occur in that component of $\textbf{w}\_{t}$ if a nonzero TD error occurs before the trade falls back to zero. The trace-decay parameter $\lambda \in \left[0, 1\right]$ determines the rate at which the trace falls.

Intuitively, they tackle the credit assignment problem by capturing both a frequency heuristic - states that are visited more often deserve more credit - and a recency heuristic - states that are visited more recently deserve more credit.

$$E\_{0}\left(s\right) = 0 $$
$$E\_{t}\left(s\right) = \gamma\lambda{E}\_{t-1}\left(s\right) + \textbf{1}\left(S\_{t} = s\right) $$

Source: Sutton and Barto, Reinforcement Learning, 2nd Edition

Source	Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com