Engineering with PyTorch

Topics covered

• Tensors
• Autograd
• Modules
• Datasets
• DataLoaders
• Optimisers
• Training loops
• GPU
• Mixed precision
• Checkpoints
• Profiling
• Memory

Lessons in this unit

1. Tensors and broadcasting 80 min
   The tensor is the building block of everything in PyTorch; learn to think in shapes, views, broadcasting and reductions using NumPy as an exact model.
2. Autograd over tensors 75 min
   From scalar autograd to PyTorch's. Every tensor operation knows how to return gradients with the shape of its inputs, and that's where broadcasting and reductions get reversed.
3. Modules, parameters and optimisers 90 min
   How nn.Module finds only your model's parameters, what state_dict stores and how SGD with momentum and AdamW update the weights, built by you in NumPy.
4. Datasets and DataLoaders 70 min
   Separating "how you get one example" from "how examples are grouped into batches". Build your Dataset, a DataLoader that's reproducible with per-epoch shuffling, and collate functions with padding.
5. A professional training loop 75 min
   The loop you'll use throughout your career. train/eval modes, gradient-free validation, early stopping with the best checkpoint, warmup with cosine decay, seeds and logging.
6. GPU, mixed precision and checkpoints 75 min
   What separates a script that trains from a training run that holds up for days on a GPU. How much memory it takes, why float16 overflows, how to accumulate gradients without changing the maths, checkpoints that really resume and how to measure before optimising.

Prerequisites

Before this unit it helps to have done:

• Deep Learning (Course 3 · Unit 1)