Activation Functions

Intuition

Stacking linear layers gives… another linear layer: $W_2(W_1x) = (W_2W_1)x$ . Non-linear activations between layers let networks bend and fold space, so they can model curves, XOR and images.

Precise explanation

name	$f(z)$	$f'(z)$
ReLU	$\max(0, z)$	1 if $z>0$ else 0
Leaky ReLU	$z$ if $z>0$ else $\alpha z$	1 if $z>0$ else $\alpha$
tanh	$\tanh z$	$1-\tanh^2 z$
sigmoid	$\sigma(z)$ (ML-11)	$\sigma(z)(1-\sigma(z))$

ReLU is not differentiable at 0; frameworks use the convention $f'(0) = 0$ , and so do we. Leaky ReLU keeps a small slope for negatives, avoiding "dead" units that never activate. tanh is zero-centred with range (−1, 1); its gradient vanishes for large $|z|$ .

Figure (original): ReLU (blue) is a hinge; tanh (red) saturates at ±1 (vertical scale differs).

Worked example

z = [−2, 0, 3]: ReLU [0, 0, 3]; ReLU′ [0, 0, 1]; leaky (α = 0.1) [−0.2, 0, 3]; tanh′(0) = 1.

Common mistakes

np.maximum(z, 0) is right; max(z, 0) (Python) fails on arrays.
Computing tanh′ as 1 - z**2 instead of 1 - tanh(z)**2.
In-place modification of the input array.

Why this matters

Activations and their derivatives are the per-layer ingredients of backpropagation (DL-10, DL-19).

References

Dive into Deep Learning — Aston Zhang, Zachary C. Lipton, Mu Li, Alexander J. Smola (S013)
CS231n: Backpropagation notes — Stanford University CS231n (S016)