The **Routing Transformer** is a [Transformer](https://paperswithcode.com/method/transformer) that endows self-attention with a sparse routing module based on online k-means. Each attention module considers a clustering of the space: the current timestep only attends to context belonging to the same cluster. In other word, the current time-step query is routed to a limited number of context through its cluster assignment.

**ScheduledDropPath** is a modified version of [DropPath](https://paperswithcode.com/method/droppath). In DropPath, each path in the cell is stochastically dropped with some fixed probability during training. In ScheduledDropPath, each path in the cell is dropped out with a probability that is linearly increased over the course of training.

ScheduledDropPath

Learning Transferable Architectures for Scalable Image Recognition

Routing Transformer

Efficient Content-Based Sparse Attention with Routing Transformers

This method proposes first discretizing observations and calculating the action distribution distance under comparable cases (intersection states).

Source	Efficient Content-Based Sparse Attention with Routing Transformers
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com