**ZeRO-Infinity** is a sharded data parallel system that extends [ZeRO](https://paperswithcode.com/method/zero) with new innovations in heterogeneous memory access called the infinity offload engine. This allows ZeRO-Infinity to support massive model sizes on limited GPU resources by exploiting CPU and NVMe memory simultaneously. In addition, ZeRO-Infinity also introduces a novel GPU memory optimization technique called memory-centric tiling to support extremely large individual layers that would otherwise not fit in GPU memory even one layer at a time.

Trainable activation function as a sigmoid-based generalization of ReLU, Swish and SiLU.

AHAF

Adaptive hybrid activation function for deep neural networks

ZeRO-Infinity

ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning

**Partition Filter Network** is a framework designed specifically for joint entity and relation extraction. The framework consists of three components: partition filter encoder, NER unit and RE unit. In task units, we use table-filling for word pair prediction. Orange, yellow and green represents NER-related, shared and RE-related component or features. (b) Detailed depiction of partition filter encoder in one single time step. We decompose feature encoding into two steps: partition and filter (shown in the gray area). In partition, we first segment neurons into two task partitions and one shared partition. Then in filter, partitions are selected and combined to form task-specific features and shared features, filtering out information irrelevant to each task.

Source	ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com