In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

**Kaiming Initialization**, or **He Initialization**, is an initialization method for neural networks that takes into account the non-linearity of activation functions, such as [ReLU](https://paperswithcode.com/method/relu) activations.

A proper initialization method should avoid reducing or magnifying the magnitudes of input signals exponentially. Using a derivation they work out that the condition to stop this happening is:

$$\frac{1}{2}n\_{l}\text{Var}\left[w\_{l}\right] = 1 $$

This implies an initialization scheme of:

$$ w\_{l} \sim \mathcal{N}\left(0,  2/n\_{l}\right)$$

That is, a zero-centered Gaussian with standard deviation of $\sqrt{2/{n}\_{l}}$ (variance shown in equation above). Biases are initialized at $0$.

Kaiming Initialization

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

AltDiffusion

AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities

**Batch Nuclear-norm Maximization** is an approach for aiding classification in label insufficient situations. It involves maximizing the nuclear-norm of the batch output matrix. The nuclear-norm of a matrix is an upper bound of the Frobenius-norm of the matrix. Maximizing nuclear-norm ensures large Frobenius-norm of the batch matrix, which leads to increased discriminability. The nuclear-norm of the batch matrix is also a convex approximation of the matrix rank, which refers to the prediction diversity.

Source	AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com