**Implicit Subspace Prior Learning**, or **ISPL**, is a framework to approach dual-blind face restoration, with two major distinctions from previous restoration methods: 1) Instead of assuming an explicit degradation function between LQ and HQ domain, it establishes an implicit correspondence between both domains via a mutual embedding space, thus avoid solving the pathological inverse problem directly. 2) A subspace prior decomposition and fusion mechanism to dynamically handle inputs at varying degradation levels with consistent high-quality restoration results.

**AdaBound** is a variant of the [Adam](https://paperswithcode.com/method/adabound) stochastic optimizer which is designed to be more robust to extreme learning rates. Dynamic bounds are employed on learning rates, where the lower and upper bound are initialized as zero and infinity respectively, and they both smoothly converge to a constant final step size. AdaBound can be regarded as an adaptive method at the beginning of training, and thereafter it gradually and smoothly transforms to [SGD](https://paperswithcode.com/method/sgd) (or with momentum) as the time step increases. 

$$ g\_{t} = \nabla{f}\_{t}\left(x\_{t}\right) $$

$$ m\_{t} = \beta\_{1t}m\_{t-1} + \left(1-\beta\_{1t}\right)g\_{t} $$

$$ v\_{t} = \beta\_{2}v\_{t-1} + \left(1-\beta\_{2}\right)g\_{t}^{2} \text{ and } V\_{t} = \text{diag}\left(v\_{t}\right) $$

$$ \hat{\eta}\_{t} = \text{Clip}\left(\alpha/\sqrt{V\_{t}}, \eta\_{l}\left(t\right), \eta\_{u}\left(t\right)\right) \text{ and } \eta\_{t} = \hat{\eta}\_{t}/\sqrt{t} $$

$$ x\_{t+1} = \Pi\_{\mathcal{F}, \text{diag}\left(\eta\_{t}^{-1}\right)}\left(x\_{t} - \eta\_{t} \odot m\_{t} \right) $$

Where $\alpha$ is the initial step size, and $\eta_{l}$ and $\eta_{u}$ are the lower and upper bound functions respectively.

AdaBound

Adaptive Gradient Methods with Dynamic Bound of Learning Rate

ISPL

Implicit Subspace Prior Learning for Dual-Blind Face Restoration

**Recurrent Dropout** is a regularization method for [recurrent neural networks](https://paperswithcode.com/methods/category/recurrent-neural-networks). [Dropout](https://paperswithcode.com/method/dropout) is applied to the updates to [LSTM](https://paperswithcode.com/method/lstm) memory cells (or [GRU](https://paperswithcode.com/method/gru) states), i.e. it drops out the input/update gate in LSTM/GRU.

Source	Implicit Subspace Prior Learning for Dual-Blind Face Restoration
Year	2000
Data Source	CC BY-SA - https://paperswithcode.com