Poem or Name Generator

You will use

• LM-03Logits, Temperature and Softmax
• LM-13Top-k and Top-p Filtering
• LM-16A Tiny GPT in PyTorch
• DL-26Saving, Loading and Reproducibility

Milestones

Build the smallest working version first. Ticks are saved in this browser only.

1. 01Collect a text file of at least 50 KB (or a list of a few thousand names, one per line).
2. 02Train the tiny GPT with a train/validation split and plot both losses.
3. 03Add temperature and top-k sampling to generate, and compare greedy with sampled output.
4. 04Scale up: more layers, a longer context, a wider model — and note the validation loss each time.
5. 05Save the best model and write a script that prints ten new samples.

Stretch goals

• Switch from characters to a BPE tokenizer (see the BPE step).
• Add a key–value cache to generation and measure the speed-up.
• Filter generated names that already appear in the training data.

Data

Run it yourself

pip install torch
python train_gpt.py --text poems.txt --steps 5000