Problem sheet

Activation Functions

activations · relu · tanh

Numerically Stable Softmax

softmax · numerical-stability · log-sum-exp

Cross-Entropy Loss

cross-entropy · loss-functions · softmax

Dense Layer Forward Pass

dense-layer · fully-connected · initialisation

PyTorch Tensors

pytorch · tensors · dtype

Models with nn.Module

pytorch · nn-module · mlp

MLP Forward PassOptional

mlp · forward-pass · cache

A Tiny Autograd Engine

autograd · computational-graph · backpropagation

Autograd in PyTorch

pytorch · autograd · backpropagation

Backprop Through One Neuron

backpropagation · chain-rule · computational-graph

Backpropagation Through a Dense Layer

backpropagation · dense-layer · relu

Custom Gradients and gradcheck

pytorch · autograd · gradient-checking

Gradient Checking by HandOptional

gradient-checking · debugging · finite-differences

SGD with Momentum

optimisers · momentum · sgd

Optimisers in torch.optim

pytorch · optimisers · adam

The Adam OptimiserOptional

optimisers · adam · adaptive-learning-rate

Datasets and DataLoader

pytorch · dataset · dataloader

A Full Training Loop

pytorch · training-loop · cross-entropy

Train an MLP on XOROptional

backpropagation · training-loop · xor

Dropout with train() and eval()

pytorch · dropout · regularisation

Inverted DropoutOptional

dropout · regularisation · train-eval-mode

Batch Normalisation with nn.BatchNorm1d

pytorch · batch-normalisation · train-eval

Batch Normalization by Hand (train/eval)Optional

batch-normalization · normalisation · running-statistics

1-D Convolution with nn.Conv1d

pytorch · convolution · conv1d

1D Convolution by Hand (valid, stride)Optional

convolution · cnn · stride

Saving, Loading and Reproducibility

pytorch · checkpoints · state-dict

Debugging Training

pytorch · debugging · gradient-clipping

Handwritten Digit Recogniser

Project idea · image-classification · pytorch · mlp

Training Diagnostics Notebook

Project idea · debugging · learning-rate · overfitting