**Kaiming Initialization**, or **He Initialization**, is an initialization method for neural networks that takes into account the non-linearity of activation functions, such as [ReLU](https://paperswithcode.com/method/relu) activations.

A proper initialization method should avoid reducing or magnifying the magnitudes of input signals exponentially. Using a derivation they work out that the condition to stop this happening is:

$$\frac{1}{2}n\_{l}\text{Var}\left[w\_{l}\right] = 1 $$

This implies an initialization scheme of:

$$ w\_{l} \sim \mathcal{N}\left(0,  2/n\_{l}\right)$$

That is, a zero-centered Gaussian with standard deviation of $\sqrt{2/{n}\_{l}}$ (variance shown in equation above). Biases are initialized at $0$.

**FORK**, or **Forward Looking Actor** is a type of actor for actor-critic algorithms. In particular, FORK includes a neural network that forecasts the next state given the current state and current action, called system network; and a neural network that forecasts the
reward given a (state, action) pair, called reward network. With the system network and reward network, FORK can forecast the next state and consider the value of the next state when improving the policy.

FORK

FORK: A Forward-Looking Actor For Model-Free Reinforcement Learning

Kaiming Initialization

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

Source	Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com