Backpropagation in One Neuron

Intuition

Backpropagation is the chain rule applied systematically: to know how the loss changes with a weight, multiply how the loss changes with the output, how the output changes with the pre-activation, and how the pre-activation changes with the weight.

Precise explanation

Graph for one example: z=w⋅x+bz = w\cdot x + b → p=σ(z)p = \sigma(z) → L=−[ylog⁡p+(1−y)log⁡(1−p)]L = -[y\log p + (1-y)\log(1-p)]. Local derivatives:

  • ∂L/∂p=−yp+1−y1−p\partial L/\partial p = -\frac{y}{p} + \frac{1-y}{1-p}
  • ∂p/∂z=p(1−p)\partial p/\partial z = p(1-p)
  • ∂z/∂w=x\partial z/\partial w = x, ∂z/∂b=1\partial z/\partial b = 1 Chain: ∂L∂z=∂L∂p∂p∂z=p−y\frac{\partial L}{\partial z} = \frac{\partial L}{\partial p}\frac{\partial p}{\partial z} = p - y, so ∇wL=(p−y)x\nabla_w L = (p-y)x and ∂L/∂b=p−y\partial L/\partial b = p-y.

Worked example

x = [1, 2], w = [0.5, −0.25], b = 0, y = 1: z = 0, p = 0.5, L = log 2 = 0.693; ∂L/∂p = −2, ∂p/∂z = 0.25, ∂L/∂z = −0.5; ∇w = [−0.5, −1], ∂b = −0.5.

Common mistakes

  • Multiplying by σ′(x)\sigma'(x) instead of σ′(z)\sigma'(z).
  • Sign errors (gradient points uphill).
  • Not reusing forward values (p) in the backward pass.

Why this matters

Every deep-learning framework does exactly this, layer by layer (DL-19).

References