Code · architecture · study mode
GPT-2 depth-12 with custom RMSNorm (no affine), GELU MLP, learned position embeddings.
Full explanation below the code →
GPT-2 depth-12 with custom RMSNorm (no affine), GELU MLP, learned position embeddings.
How many parameters in the d12 preset? What norm is used?
Baseline step — no previous snapshot to diff against.
Navigate by roadmap
| Step | Topic | This lesson |
|---|---|---|
| Getting started | Welcome to FBA Lab | — |
| #1 Baseline | The baseline training run | You are here |
| #2.1 Architecture Tweaks | RoPE, ReLU² MLP, scaled attention output, torch.compile. | Coming soon |
After you finish reading, use the code panel above to walk the four blocks — GPT model, DistributedDataLoader, Training loop, and DDP setup — and match each one to the diagram.