This method proposes first discretizing observations and calculating the action distribution distance under comparable cases (intersection states).

The **Routing Transformer** is a [Transformer](https://paperswithcode.com/method/transformer) that endows self-attention with a sparse routing module based on online k-means. Each attention module considers a clustering of the space: the current timestep only attends to context belonging to the same cluster. In other word, the current time-step query is routed to a limited number of context through its cluster assignment.

Routing Transformer

Efficient Content-Based Sparse Attention with Routing Transformers

Playstyle Distance

An Unsupervised Video Game Playstyle Metric via State Discretization

**Sarsa** is an on-policy TD control algorithm:

$$Q\left(S\_{t}, A\_{t}\right) \leftarrow Q\left(S\_{t}, A\_{t}\right) + \alpha\left[R_{t+1} + \gamma{Q}\left(S\_{t+1}, A\_{t+1}\right) - Q\left(S\_{t}, A\_{t}\right)\right] $$

This update is done after every transition from a nonterminal state $S\_{t}$. if $S\_{t+1}$ is terminal, then $Q\left(S\_{t+1}, A\_{t+1}\right)$ is defined as zero.

To design an on-policy control algorithm using Sarsa, we estimate $q\_{\pi}$ for a behaviour policy $\pi$ and then change $\pi$ towards greediness with respect to $q\_{\pi}$.

Source: Sutton and Barto, Reinforcement Learning, 2nd Edition

Source	An Unsupervised Video Game Playstyle Metric via State Discretization
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com