A novel built-in attention mechanism, that is complementary to all other prior attention mechanisms (e.g. squeeze and excitation, transformers) that are external (i.e., not built-in - please read paper for more details)

**PipeTransformer** is a method for automated elastic pipelining for efficient distributed training of [Transformer](https://paperswithcode.com/method/transformer) models. In PipeTransformer, an adaptive on the fly freeze algorithm is used that can identify and freeze some layers gradually during training, as well as an elastic pipelining system that can dynamically allocate resources to train the remaining active layers. More specifically, PipeTransformer automatically excludes frozen layers from the pipeline, packs active layers into fewer GPUs, and forks more replicas to increase data-parallel width.

PipeTransformer

PipeTransformer: Automated Elastic Pipelining for Distributed Training of Transformers

Weight excitation

Weight Excitation: Built-in Attention Mechanisms in Convolutional Neural Networks

**MushroomRL** is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. The architecture of MushroomRL is built in such a way that every component of an RL problem is already provided, and most of the time users can only focus on the implementation of their own algorithms and experiments. MushroomRL comes with a strongly modular architecture that makes it easy to understand how each component is structured and how it interacts with other ones; moreover it provides an exhaustive list of RL methodologies, such as:

Source	Weight Excitation: Built-in Attention Mechanisms in Convolutional Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com