The simplest code that completes a training run
This section presents the simplest code that can complete a training run. This is the perfect way to start learning — building from simpler examples instead of looking at complex engineering code directly. The best part is that once you go through this basic script, a lot of the concepts and learning are very transferable. Attention, QK-norm, residual connections, learning rate — many concepts transfer to scaled experiments.
Scaling & its implication
AI training, at its core, is a game of scaling large systems. This field can overwhelm a lot of people, and the fact that most engineers and teams don't have access to GPUs can be a big barrier to learning. There is a lot of content and code provided by teams who have built large models, and we can learn large-systems engineering using them. Instead of getting overwhelmed, we will anchor our learning with the speedrun and use simulation before we can fire GPUs. These will be actual runs, simulated for learning, with a lot of visualization.
Three acts, not one competition
Instead of focusing on the nanoGPT speedrun competition — which is a popular competition — we will divide our journey into 3 acts.
Act 1 will be a simple code/script. We can start with something simple and build onto it over 6 iterations, understanding the impact of each change. This creates an incubator for you to get familiar with a lot of concepts, terms, and the intuition behind them. As an example, you will get introduced to gradient accumulation and learning rate, and how and when model updates are done. It also creates a perfect entry point to learn deeper concepts — a nice, organized path to learning. You can refer to this path by looking at the roadmap, as well as the main speedrun, which is neatly divided into beats.
Act 2 is where you look at the profiler and low-level optimization. Profiler traces are the most fun to look at and give insights into actual performance under the hood.
Act 3 is the actual speedrun, or part of it. There are now 80+ iterations of this code, with a lot of participation from AI engineers across the world. We will look at a few of the iterations.
The skill you're building: LLM Training & Systems Engineering
Earned across three acts — not by memorizing facts, but by reading code, tracing losses, and making one decision at a time. By the end it's a resume line you can defend, because you'll have coded it.
Every architecture change is a single diff. Understand what changed before asking why it worked.
Intuition follows from watching the curve move. Theory explains after the fact.
The speedrun works because each step changes exactly one thing. Isolate your bets.
You'll be able to explain how to read a training script and what each change buys — RoPE, the Muon optimizer, dataloading, softcap, and long context — read off the val-loss curve.
You'll be able to explain how to profile a real H100 training run with Perfetto and find the bottleneck.
You'll be able to explain the record-breaking training PRs — Flash Attention 3, FP8, Polar Express, and multi-token prediction.
Distributed training (DDP · torchrun · Modal) and the full FBA Lab Speedrun — profiler traces, W&B metrics, and every step you walk here. Clone it and run it.
We trained GPT-2 from scratch in 23 minutes
Today we ran the nanoGPT speedrun experiment on 8× NVIDIA H100 GPUs. Below is what we set out to do, how the experiment is scored, and the baseline result every future beat will try to beat. This run is Beat 01 — the baseline — and it's the benchmark you'll try to improve on across the next five iterations of Act 1.
Find the best GPT-2 model — the architecture and training recipe that reaches a good model the fastest on a given dataset.
Concretely: we train a GPT-2 small (124M parameter) model from scratch until it reaches a validation loss below 3.28, and we record how long that took. Then, in each following iteration, we try to improve that time by making architectural or optimization changes to the model — same target, faster every step.
The baseline result
huggingface.co/datasets/kjj0/fineweb10B-gpt2 — contains 10 billion tokens from FineWeb converted into a pre-tokenized binary format. Our training run consumed 6.44 billion tokens and took 23 minutes to complete. This is the baseline we will try to improve in the next beat.
To go deeper, open the first lab: Baseline run, decoded →
Built the way top labs actually train models
The FBA Lab speedrun shows how a training run gets faster — one real decision at a time. Each step appears on the loss curve so you can see why teams iterate on code before they chase bigger models or more GPUs.
Why run this lab? You read actual training scripts, walk through what changed between steps, and build the same intuition engineers use when shaving hours off a run — in a guided, visual format instead of hunting through repos and papers alone.
Loss values and speed-ups are illustrative — simplified to teach the shape of the idea, not exact benchmark numbers. Click a step below the chart to jump into that lab in the full speedrun journey.
Same run, two machines
This is the most important table in Act 1 — it's your baseline run and time, and the number every later iteration is measured against. The exact same baseline GPT-2 training run — target val loss 3.28, 262,144 tokens/step — on a cloud H100 node vs. a two-GPU desktop. Tyler's original speedrun ran on smaller hardware, 2× RTX 4090; we run the same recipe on 8× H100.
| Hardware | grad accum | Wall-clock | Steps | Tokens | tok/s | |
|---|---|---|---|---|---|---|
| 8× NVIDIA H100 | 1 | 24,576 | ≈6.44B† | ≈4.6M† | Study it → | |
| 2× RTX 4090 | 4 | 24,576 | 6.44B | 221k |
≈21× faster wall-clock on the 8× H100 node — same recipe, same tokens.
† derived: tokens = steps × 262,144; H100 tok/s = tokens ÷ 1,400 s.
The 2× RTX 4090 speedrun — 6 iterations
| # | Iteration | Time | Steps | Tokens | tok/s | |
|---|---|---|---|---|---|---|
| 1 | Initial baseline | 24,576 | 6.44B | 221k | Study it → | |
| 2 | Architectural changes | 9,664 | 5.07B | 188k | Study it → | |
| 3 | Muon optimizer | 5,800 | 3.04B | 187k | Study it → | |
| 4 | Dataloading tweaks | 6,312 | 3.31B | 216k | Study it → | |
| 5 | Logit soft-capping @ 30 | 6,120 | 3.15B | 218k | Study it → | |
| 6 | Longer sequence length | 3,584 | 1.88B | 205k | Study it → |
- Times & tokens are Tyler's reported records at ~3.28 val loss.
- Steps are the num_iterations budgets in this repo's step folders (01…06); runs can stop earlier once val loss ≤ 3.28.
- Baseline uses grad_accum_steps=4 on 2 GPUs to keep the global batch at 262,144 tokens/step.
Speedrun on 2× RTX 4090 by Tyler Romero · worklog
From 45 minutes to 1.65 minutes
This is Act 3 — the official speedrun competition code, run for real. Its baseline took 45 minutes; by record #63 it was down to 1.650 minutes. At that point, provisioning the GPU takes longer than the run itself. The record-breaking modded-nanogpt PRs you study here — each a real speed record on 8× H100, training to the same 3.28 val loss.
| # | What changed | Record time | FBA Lab lesson | PR | |
|---|---|---|---|---|---|
| #1 | llm.c GPT-2 baseline — the start line | 45 min | llm.c GPT-2 baseline | llm.c | |
| #29 | Flash Attention 3, 2048 max_doc_len, updated window schedule | 2.731 min | The attention bottleneck lesson | #118 | Study it → |
| #38 | Polar Express — a replacement for Newton-Schulz | 2.476 min | Optimizer internals beyond AdamW | #134 | Study it → |
| #46 | Batch size schedule | 2.203 min | Token efficiency vs GPU efficiency | #163 | Study it → |
| #53 | Multi-token prediction, untie embed/lm_head at 2/3 training | 1.988 min | More learning signal per forward pass | #178 | Study it → |
| #54 | Asymmetric logit rescale | 1.940 min | Output head stability and loss shaping | #181 | Study it → |
| #62 | Bigram hash embedding | 1.655 min | A clever n-gram prior inside a transformer | #201 | Study it → |
| #63our run | Untie value embeds | 1.650 min | Untie Value Embeds — the run we did | #209 | |
| #84 | FP8 — hardware-aware precision (a later record) | — | Hardware-aware precision | #306 | Study it → |
Source: modded-nanogpt: Speedrunning the NanoGPT baseline — Keller Jordan, Jeremy Bernstein, Brendan Rappazzo, et al. (2024) · KellerJordan/modded-nanogpt
Having understood the practical limitations of this setup, we can now be clear about what we can learn.
- The biggest win is LLM modelling. Understanding the major moving parts of a model, and how architecture and changes affect a training run, is the biggest win.
- Another win is DDP and PyTorch. If you are a coder or interested in math, learners from both backgrounds will get actionable code, blogs, and visualization to ground their learning. So if you are looking to break into AI, this is the perfect start.
- Some topics in AI training are very transferable and even make you more skilled in other areas. For example, inference, sampling, and MTP can be understood even better if you have done iterations of LLM architectural changes. Tokenization can be studied in depth — the same goes for position embeddings, context length, and attention. Even reasoning and AI agentic behaviour can be understood if you understand AI training.
- You also build a habit of benchmarking — set a target, measure a run honestly, and compare against a baseline before drawing conclusions. That “define the metric, measure, compare” discipline is a skill that carries well outside AI training, into any performance or systems work.
- Further in, you get into low-level optimization and kernel development — profiling where the time actually goes, writing and tuning custom kernels, and seeing how code maps onto the hardware. These systems skills transfer to any performance-critical engineering, not just model training.
Even with imagination, content, and other superpowers, some things are out of reach here.
- MoE (mixture of experts), CoT reasoning training, and agentic tool use. Modern frontier models use different combinations of these — and this speedrun does not cover them.
- This setup only deals with DDP, or 1D. Most modern pipelines work with 4D and more parallelism.
- The biggest problem with this setup is that we don't get a usable checkpoint. We can only look at loss and graphs. So if inference is your main goal, you might miss out on some of the things.
- Data pipeline engineering is also oversimplified — actual setups have complex data pipelines.