Smish is an activation function defined as $f(x)=x\cdot \text{tanh}(\ln(1+\sigma(x)))$ where $\sigma(x)$ denotes the sigmoid function. A parameterized version was also described in the form $f(x)=\alpha x\cdot \text{tanh}(\ln(1+\sigma(\beta x)))$.

Paper: Smish: A Novel Activation Function for Deep Learning Methods

Source: https://www.mdpi.com/2079-9292/11/4/540

**MobileNet** is a type of convolutional neural network designed for mobile and embedded vision applications. They are based on a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks that can have low latency for mobile and embedded devices.

MobileNetV1

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Smish

**SimCLR** is a framework for contrastive learning of visual representations. It learns representations by maximizing agreement between differently augmented views of the same data example via a contrastive loss in the latent space. It consists of:

- A stochastic data augmentation module that transforms any given data example randomly resulting in two correlated views of the same example, denoted $\mathbf{\tilde{x}\_{i}}$ and $\mathbf{\tilde{x}\_{j}}$, which is considered a positive pair. SimCLR sequentially applies three simple augmentations: random cropping followed by resize back to the original size, random color distortions, and [random Gaussian blur](https://paperswithcode.com/method/random-gaussian-blur). The authors find random crop and color distortion is crucial to achieve good performance.

- A neural network base encoder $f\left(·\right)$ that extracts representation vectors from augmented data examples. The framework allows various choices of the network architecture without any constraints. The authors opt for simplicity and adopt [ResNet](https://paperswithcode.com/method/resnet) to obtain $h\_{i} = f\left(\mathbf{\tilde{x}}\_{i}\right) = \text{ResNet}\left(\mathbf{\tilde{x}}\_{i}\right)$ where $h\_{i} \in \mathbb{R}^{d}$ is the output after the [average pooling](https://paperswithcode.com/method/average-pooling) layer.

- A small neural network projection head $g\left(·\right)$ that maps representations to the space where contrastive loss is applied. Authors use a MLP with one hidden layer to obtain $z\_{i} = g\left(h\_{i}\right) = W^{(2)}\sigma\left(W^{(1)}h\_{i}\right)$ where $\sigma$ is a [ReLU](https://paperswithcode.com/method/relu) nonlinearity. The authors find it beneficial to define the contrastive loss on $z\_{i}$’s rather than $h\_{i}$’s.

- A contrastive loss function defined for a contrastive prediction task. Given a set {$\mathbf{\tilde{x}}\_{k}$} including a positive pair of examples $\mathbf{\tilde{x}}\_{i}$ and $\mathbf{\tilde{x}\_{j}}$ , the contrastive prediction task aims to identify $\mathbf{\tilde{x}}\_{j}$ in {$\mathbf{\tilde{x}}\_{k}$}$\_{k\neq{i}}$ for a given $\mathbf{\tilde{x}}\_{i}$.

A minibatch of $N$ examples is randomly sampled and the contrastive prediction task is defined on pairs of augmented examples derived from the minibatch, resulting in $2N$ data points. Negative examples are not sampled explicitly. Instead, given a positive pair, the other $2(N − 1)$ augmented examples within a minibatch are treated as negative examples. A [NT-Xent](https://paperswithcode.com/method/nt-xent) (the normalized
temperature-scaled cross entropy loss) loss function is used (see components).

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com