Parameters, gradients and optimizer state spread across cards, with what PyTorch ships.
Before this guide: Accelerate, Deep Dive.
The notebooks
The outline for this guide is not written yet. It will name every notebook, in reading order, before the first one is written.