The Adam Optimiser

Intuition

DL-14 smooths the direction of the gradient. Adam also adapts the step size of each parameter: parameters whose gradients are consistently large take proportionally smaller steps, and rarely-updated parameters take larger ones. It is the default optimiser for most deep learning.

Precise explanation

For each parameter, keep mm (moving average of gradients) and vv (moving average of squared gradients), both starting at 0, and a step counter tt shared by all parameters. At step t=1,2,…t = 1, 2, \dots:

m←β1m+(1−β1)g,v←β2v+(1−β2)g2,m \leftarrow \beta_1 m + (1-\beta_1) g,\qquad v \leftarrow \beta_2 v + (1-\beta_2) g^2, m^=m1−β1t,v^=v1−β2t,θ←θ−lr m^v^+ϵ.\hat m = \frac{m}{1-\beta_1^t},\qquad \hat v = \frac{v}{1-\beta_2^t},\qquad \theta \leftarrow \theta - \text{lr}\,\frac{\hat m}{\sqrt{\hat v} + \epsilon}.
  • Bias correction: because mm and vv start at 0 they are too small early on; dividing by 1−βt1-\beta^t fixes that. At t=1t = 1, m^=g\hat m = g and v^=g2\hat v = g^2, so the first step is lr⋅g/(∣g∣+ϵ)≈lr⋅sign⁡(g)\text{lr}\cdot g/(|g| + \epsilon) \approx \text{lr}\cdot\operatorname{sign}(g).
  • Multiplying all gradients by a constant leaves the steps (almost) unchanged — Adam is scale-invariant, unlike plain SGD.
  • Defaults from the paper: β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}.

Worked example

g=4g = 4, lr =0.1= 0.1, first step: m=0.4m = 0.4, v=0.016v = 0.016, m^=4\hat m = 4, v^=16\hat v = 16, step =0.1⋅4/(4+10−8)≈0.1= 0.1 \cdot 4/(4 + 10^{-8}) \approx 0.1.

Common mistakes

  • Incrementing tt once per parameter instead of once per step call.
  • Forgetting bias correction, giving tiny early steps.
  • Putting ϵ\epsilon inside the square root (a different, but common, variant; the paper puts it outside).

Why this matters

Adam (and its variant AdamW) trains most transformers and CNNs; knowing its update explains learning-rate choices and warm-up schedules.

Library implementation

torch.optim.Adam(parameters, lr=..., betas=(0.9, 0.999), eps=1e-8) implements the same update (with ε outside the square root). In PyTorch the optimiser modifies tensors in place after loss.backward(); here the dictionary interface makes each step explicit. The comparison below runs only when PyTorch is installed.

References