**Gaussian Processes** are non-parametric models for approximating functions. They rely upon a measure of similarity between points (the kernel function) to predict the value for an unseen point from training data. The models are fully probabilistic so uncertainty bounds are baked in with the model.

Image Source: Gaussian Processes for Machine Learning, C. E. Rasmussen & C. K. I. Williams

**Minimum Description Length** provides a criterion for the selection of models, regardless of their complexity, without the restrictive assumption that the data form a sample from a 'true' distribution.

Extracted from [scholarpedia](http://scholarpedia.org/article/Minimum_description_length)

**Source**:

Paper: [J. Rissanen (1978) Modeling by the shortest data description. Automatica 14, 465-471](https://doi.org/10.1016/0005-1098(78)90005-5)

Book: [P. D. Grünwald (2007) The Minimum Description Length Principle, MIT Press, June 2007, 570 pages](https://ieeexplore.ieee.org/servlet/opac?bknumber=6267274)

Gaussian Process

**DINO** (self-distillation with no labels) is a self-supervised learning method that directly predicts the output of a teacher network - built with a momentum encoder - using a standard cross-entropy loss. 

In the example to the right, DINO is illustrated in the case of one single pair of views $\left(x\_{1}, x\_{2}\right)$ for simplicity.
The model passes two different random transformations of an input image to the student and teacher networks. Both networks have the same architecture but other parameters.
The output of the teacher network is centered with a mean computed over the batch. Each network outputs a $K$ dimensional feature normalized with a temperature [softmax](https://paperswithcode.com/method/softmax) over the feature dimension.
Their similarity is then measured with a cross-entropy loss.
A stop-gradient (sg) operator is applied to the teacher to propagate gradients only through the student.
The teacher parameters are updated with the student parameters' exponential moving average (ema).

Year	2000
Data Source	CC BY-SA - https://paperswithcode.com