Derivative Rules and the Chain Rule

Intuition

MA-10 estimates a derivative by nudging the input. Rules give the exact derivative of a formula built from simpler pieces. The most important is the chain rule: when one function feeds another, their rates of change multiply — the idea behind backpropagation.

Precise explanation

Rule Function Derivative
power xnx^n nxn−1n x^{n-1}
sum f+gf + g f′+g′f' + g'
product fgf g f′g+fg′f' g + f g'
chain f(g(x))f(g(x)) f′(g(x)) g′(x)f'(g(x))\, g'(x)
exp, log exe^x, ln⁡x\ln x exe^x, 1/x1/x
sigmoid σ(x)\sigma(x) σ(x)(1−σ(x))\sigma(x)(1-\sigma(x))

In code, represent a differentiable function as a pair (f, df) of Python functions. Each rule takes pairs and returns a new pair, so you can build derivatives of complicated formulas from small, tested pieces.

def chain_rule(f, df, g, dg):
    h = lambda x: f(g(x))
    dh = lambda x: df(g(x)) * dg(x)
    return h, dh

Worked example

h(x)=σ(3x+1)h(x) = \sigma(3x + 1): outer f=σf = \sigma, inner g(x)=3x+1g(x) = 3x + 1 with g′=3g' = 3, so h′(x)=σ(3x+1)(1−σ(3x+1))⋅3h'(x) = \sigma(3x+1)(1-\sigma(3x+1)) \cdot 3. At x=0x = 0: σ(1)≈0.7311\sigma(1) \approx 0.7311 and h′(0)≈0.7311⋅0.2689⋅3≈0.5898h'(0) \approx 0.7311 \cdot 0.2689 \cdot 3 \approx 0.5898.

Common mistakes

  • Forgetting the inner derivative: ddxσ(3x)≠σ′(3x)\frac{d}{dx}\sigma(3x) \ne \sigma'(3x); multiply by 3.
  • Evaluating the outer derivative at xx instead of at g(x)g(x).
  • Product rule as f′g′f' g'.

Why this matters

Backpropagation (DL-10, DL-08) is the chain rule applied repeatedly, and gradient descent (MA-13) needs these derivatives.

References