Partial Derivatives and the Gradient

Intuition

A function of several variables is a landscape. The partial derivative with respect to xix_i is the slope if you walk only along the xix_i axis. Collecting all partials gives the gradient — an arrow pointing in the direction of steepest ascent.

Precise explanation

For f:Rn→Rf: \mathbb{R}^n \to \mathbb{R},

∇f(x)=(∂f∂x1,…,∂f∂xn),∂f∂xi≈f(x+hei)−f(x−hei)2h,\nabla f(x) = \left(\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n}\right),\qquad \frac{\partial f}{\partial x_i} \approx \frac{f(x + h e_i) - f(x - h e_i)}{2h},

where eie_i is the unit vector along axis ii (perturb one coordinate, keep the others fixed). This needs 2n2n function evaluations.

Gradient descent in nn dimensions: x←x−η∇f(x)x \leftarrow x - \eta \nabla f(x).

Worked example

f(x,y)=x2+3xyf(x, y) = x^2 + 3xy. Exact: ∂f/∂x=2x+3y\partial f/\partial x = 2x + 3y, ∂f/∂y=3x\partial f/\partial y = 3x. At (1,2)(1, 2): ∇f=(8,3)\nabla f = (8, 3).

Common mistakes

  • Perturbing the original list in place and forgetting to restore it — later partials are wrong.
  • Perturbing all coordinates at once (that gives a directional derivative along (1,…,1)).
  • Returning a tuple of tuples instead of a flat list of partials.

Why this matters

Numerical gradients are how you check backpropagation (DL-13). Understanding "one coordinate at a time" also explains why numerical gradients are too slow to train networks with millions of parameters, motivating automatic differentiation.

References