Your AI model training roadmap
The FBA Lab Speedrun is a hands-on journey into how modern AI models are trained — in real Python and PyTorch. Every idea is grounded in real training code, explored through optional deep dives, and compared to how frontier open models — OLMo, Marin, SmolLM3 — actually do it. One skill: LLM Training & Systems Engineering. FBA Lab is free — no login, no payment.
Welcome to FBA Lab
In FBA Lab you'll learn how to build real AI — by training LLM models across multiple GPUs. You'll grow fluency in the topics that matter and build skills that count, working with real code, logs, visualizations, and Weights & Biases charts. Every step pairs a video and written lesson with an optional deep dive.
- Why training a model needs multiple GPUs in the first place
- What a speedrun experiment is really aiming for — and the skill of improving a model's architecture on each iteration of the run
- The dataset used to train the model, and why it's stored in that format
- In the Welcome beat: what distributed data processing is, and how a training run works in 1-D data parallelism
The baseline training run
This is where the real intuition clicks — the biggest and most interesting part of FBA Lab. You read and edit the actual GPT-2 training script and watch one full run, end to end.
- The one training loop that never changes — load a batch → forward → backward → update — and why reading it is what an AI engineer actually does
- The parts that make up a GPT-2 model, and how 8 GPUs train one identical copy together (DDP)
- Where the data comes from — pre-tokenized shards and a loader that keeps all 8 GPUs fed with different text
- How a weight actually changes — clip the gradient, then AdamW takes the step (Muon comes in a later step)
- The learning-rate triangle (warm up, then coast down) and reading your Weights & Biases charts: train vs validation loss, grad-norm, learning rate
- The benchmark you're chasing: target val-loss 3.28 on 8× H100 — the baseline hits it in ~23 minutes over 6.44B tokens
How to feed data into the model correctly, how the model updates after seeing that data, and how the pieces of training work in PyTorch — how a script holds a large model in memory and the critical parts of the training loop across the forward and backward pass (for example, how gradients flow and weights change).
The skills you walk away with
- The biggest win is LLM modelling. Understanding the major moving parts of a model, and how architecture and changes affect a training run, is the biggest win.
- Another win is DDP and PyTorch. If you are a coder or interested in math, learners from both backgrounds will get actionable code, blogs, and visualization to ground their learning. So if you are looking to break into AI, this is the perfect start.
- Some topics in AI training are very transferable and even make you more skilled in other areas. For example, inference, sampling, and MTP can be understood even better if you have done iterations of LLM architectural changes. Tokenization can be studied in depth — the same goes for position embeddings, context length, and attention. Even reasoning and AI agentic behaviour can be understood if you understand AI training.
- You also build a habit of benchmarking — set a target, measure a run honestly, and compare against a baseline before drawing conclusions. That “define the metric, measure, compare” discipline is a skill that carries well outside AI training, into any performance or systems work.
- Further in, you get into low-level optimization and kernel development — profiling where the time actually goes, writing and tuning custom kernels, and seeing how code maps onto the hardware. These systems skills transfer to any performance-critical engineering, not just model training.
Frequently asked questions
Is FBA Lab free?
Yes — FBA Lab is completely free. Every lesson, video, and deep dive is free to read and watch, with no login and no payment.
I work full-time — what's the time commitment?
Around 6 hours a week at most. I know you're busy — so am I. Lessons are self-paced, so you can move faster or slower without falling behind.
Can I revisit lessons any time?
Yes. Every lesson beat is recorded, and each one comes with a text lesson and visualizations you can revisit any time. The deep dives are the real gems — detailed lessons that go deep on the crucial topics, with both video and text.