**Parameterized Exponential Linear Units**, or **PELU**, is an activation function for neural networks. It involves learning a parameterization of [ELU](https://paperswithcode.com/method/elu) in order to learn the proper activation shape at each layer in a CNN. 

The PELU has two additional parameters over the ELU:

$$ f\left(x\right) = cx \text{ if } x > 0 $$
$$ f\left(x\right) = \alpha\exp^{\frac{x}{b}} - 1 \text{ if } x \leq 0 $$

Where $a$, $b$, and $c > 0$. Here $c$ causes a change in the slope in the positive quadrant, $b$ controls the scale of the [exponential decay](https://paperswithcode.com/method/exponential-decay), and $\alpha$ controls the saturation in the negative quadrant.

Source: [Activation Functions](https://arxiv.org/pdf/1811.03378.pdf)

A **Class Attention** layer, or **CA Layer**, is an [attention mechanism](https://paperswithcode.com/methods/category/attention-mechanisms-1) for [vision transformers](https://paperswithcode.com/methods/category/vision-transformer) used in [CaiT](https://paperswithcode.com/method/cait) that aims to extract information from a set of processed patches. It is identical to a [self-attention layer](https://paperswithcode.com/method/scaled), except that it relies on the attention between (i) the class embedding $x_{\text {class }}$ (initialized at CLS in the first CA) and (ii) itself plus the set of frozen patch embeddings $x_{\text {patches }} .$ 

Considering a network with $h$ heads and $p$ patches, and denoting by $d$ the embedding size, the multi-head class-attention is parameterized with several projection matrices, $W_{q}, W_{k}, W_{v}, W_{o} \in \mathbf{R}^{d \times d}$, and the corresponding biases $b_{q}, b_{k}, b_{v}, b_{o} \in \mathbf{R}^{d} .$ With this notation, the computation of the CA residual block proceeds as follows. We first augment the patch embeddings (in matrix form) as $z=\left[x_{\text {class }}, x_{\text {patches }}\right]$. We then perform the projections:

$$Q=W\_{q} x\_{\text {class }}+b\_{q}$$

$$K=W\_{k} z+b\_{k}$$

$$V=W\_{v} z+b\_{v}$$

The class-attention weights are given by

$$
A=\operatorname{Softmax}\left(Q . K^{T} / \sqrt{d / h}\right)
$$

where $Q . K^{T} \in \mathbf{R}^{h \times 1 \times p}$. This attention is involved in the weighted sum $A \times V$ to produce the residual output vector

$$
\operatorname{out}\_{\mathrm{CA}}=W\_{o} A V+b\_{o}
$$

which is in turn added to $x\_{\text {class }}$ for subsequent processing.

Class Attention

Going deeper with Image Transformers

PELU

Parametric Exponential Linear Unit for Deep Convolutional Neural Networks

**Capsule Network** is a machine learning system that is a type of artificial neural network that can be used to better model hierarchical relationships. The approach is an attempt to more closely mimic biological neural organization.

Source	Parametric Exponential Linear Unit for Deep Convolutional Neural Networks
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com