What you're actually here to build

One skill: LLM Training & Systems Engineering — the ability to open the hood on a real training run and *follow* it. Not "watch a course," not "memorize technique names." By the end you'll be able to read how record-breaking models are actually trained the way you already read a pull request: look at the change, check the test, form an opinion.

Here's the concrete setup we'll use for the whole journey: the goal is to learn how to train a large language model across multiple GPUs, and we'll do it at a scale you can actually reason about — 8 GPUs, the same order of magnitude real labs warm up on before scaling further. Take one model, put an identical copy of it on each of the 8 GPUs, and number the GPUs 0 through 7. That number is the GPU's rank — GPU 0 is rank 0, and by convention it's the primary (or "master") GPU that coordinates the others. Every rank starts out holding the exact same model weights.

Figure — 8 GPUs, numbered 0–7. That number is the rank. Rank 0 is the primary GPU; every rank holds an identical copy of the model.

Why do we need 8 GPUs? Speed. Hand a large dataset to a single GPU and it will train — one batch at a time, however long that takes. Split that same data into 8 pieces and hand one piece to each of the 8 GPUs instead, and all 8 train in parallel: the same amount of data gets processed in a fraction of the time. This is Distributed Data Parallel (DDP) — the model is replicated on every rank, each rank crunches a different slice of the batch, and the ranks sync their gradients back together every step.

This is the entry-level move in distributed training. As models and clusters get bigger, labs reach for richer forms of parallelism — 2D, 3D, even 4D — splitting not just the data but the model's layers, tensors, and pipeline stages across GPUs too, each with its own techniques and tooling. Because this journey only splits the data, we'll call what you're learning here 1D: one axis of parallelism, and the foundation everything else is built on.

Want the code-level version? DDP in plain Python walks through it line by line.

How the journey is built

The whole thing is a real distributed training speedrun — an 8×H100 run — decompiled into moves you can inspect one at a time. It runs in three acts:

• Act 1 · Fundamentals — a baseline `train_gpt2.py`, then six one-change diffs that walk the loss from 10.99 down toward the target of 3.28.
• Act 2 · Systems — a real H100 profiler trace: where the time actually goes.
• Act 3 · World records — the actual nanoGPT speedrun milestone PRs, read as code review.

The loss curve is your scoreboard and the three acts are your "where am I." Every step is, in effect, a pull request: a diff, a reason, and a green/red test.

Every step comes three ways

You choose how deep to go on each step — the goal is the skill, not the checkbox:

• ▶ Watch — a short video lesson frames what changed and why.
• ¶ Read — a written lesson you can take at your own pace.
• ✦ Deep Dive — optional deep dives on the concepts underneath, triggered only when you're curious. These span the whole skill, not just one beat.

How to start

Watch the video, skim this lesson, then continue to #1 Baseline when you're ready. You don't have to use all three modes on every step — use whichever gets the idea into your head.