Trainable activation function as a sigmoid-based generalization of ReLU, Swish and SiLU.

**Capsule Network** is a machine learning system that is a type of artificial neural network that can be used to better model hierarchical relationships. The approach is an attempt to more closely mimic biological neural organization.

CapsNet

Dynamic Routing Between Capsules

AHAF

Adaptive hybrid activation function for deep neural networks

**ZeRO-Infinity** is a sharded data parallel system that extends [ZeRO](https://paperswithcode.com/method/zero) with new innovations in heterogeneous memory access called the infinity offload engine. This allows ZeRO-Infinity to support massive model sizes on limited GPU resources by exploiting CPU and NVMe memory simultaneously. In addition, ZeRO-Infinity also introduces a novel GPU memory optimization technique called memory-centric tiling to support extremely large individual layers that would otherwise not fit in GPU memory even one layer at a time.

Source	Adaptive hybrid activation function for deep neural networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com