**Hydra** is a multi-headed neural network for model distillation with a shared body network. The shared body network learns a joint feature representation that enables each head to capture the predictive behavior of each ensemble member.  Existing distillation methods often train a distillation network to imitate the prediction of a larger network. Hydra instead learns to distill the individual predictions of each ensemble member into separate light-weight head models while amortizing the computation through a shared heavy-weight body network. This retains the diversity of ensemble member predictions which is otherwise lost in knowledge distillation.

**NADAM**, or **Nesterov-accelerated Adaptive Moment Estimation**, combines [Adam](https://paperswithcode.com/method/adam) and [Nesterov Momentum](https://paperswithcode.com/method/nesterov-accelerated-gradient). The update rule is of the form:

$$ \theta\_{t+1} = \theta\_{t} - \frac{\eta}{\sqrt{\hat{v}\_{t}}+\epsilon}\left(\beta\_{1}\hat{m}\_{t} + \frac{(1-\beta\_{t})g\_{t}}{1-\beta^{t}\_{1}}\right)$$

Image Source: [Incorporating Nesterov Momentum into Adam](http://cs229.stanford.edu/proj2015/054_report.pdf)

NADAM

Hydra

Hydra: Preserving Ensemble Diversity for Model Distillation

**Variational Trace Distance Estimation**, or **VTDE**, is a variational algorithm for trace norm estimation that only involves one ancillary qubit. Notably, the cost function in VTDE gathers information from a single-qubit observable and thus could avoid the barren plateau issue with logarithmic depth parameterized circuits.

Source	Hydra: Preserving Ensemble Diversity for Model Distillation
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com