**Linear Warmup With Cosine Annealing** is a learning rate schedule where we increase the learning rate linearly for $n$ updates and then anneal according to a cosine schedule afterwards.

A **Hermite Activations** is a type of activation function which uses a smooth finite Hermite polynomial base as a substitute for non-smooth [ReLUs](https://paperswithcode.com/method/relu). 

Relevant Paper: [Lokhande et al](https://arxiv.org/pdf/1909.05479.pdf)

Hermite Activation

Linear Warmup With Cosine Annealing

A GNN for dags, which injects their topological order as an inductive bias via asynchronous message passing.

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com