Viet-Anh on Software Logo

What is: Forward gradient?

SourceGradients without Backpropagation
Year2000
Data SourceCC BY-SA - https://paperswithcode.com

Forward gradients are unbiased estimators of the gradient f(θ)\nabla f(\theta) for a function f:RnRf: \mathbb{R}^n \rightarrow \mathbb{R}, given by g(θ)=f(θ),vvg(\theta) = \langle \nabla f(\theta) , v \rangle v.

Here v=(v1,,vn)v = (v_1, \ldots, v_n) is a random vector, which must satisfy the following conditions in order for g(θ)g(\theta) to be an unbiased estimator of f(θ)\nabla f(\theta)

  • vivjv_i \perp v_j for all iji \neq j
  • E[vi]=0\mathbb{E}[v_i] = 0 for all ii
  • V[vi]=1\mathbb{V}[v_i] = 1 for all ii

Forward gradients can be computed with a single jvp (Jacobian Vector Product), which enables the use of the forward mode of autodifferentiation instead of the usual reverse mode, which has worse computational characteristics.