Trace Journey · The real speedrun
What the Profiler Reveals
The Perfetto trace from the speedrun captured 3 fully-compiled training steps. Six engineering choices show up in those frames that are unique to this architecture — from a hash-collision fix that won a world record to a 12-parameter gate that blends neighboring tokens. This is what they do — and why big production LLMs don't need them.
// six techniques in this group
- 01get_bigram_hash + sign trick — collision fix that set the WR (79.7s)
- 02polar_express — orthogonalizing gradients for NorMuon
- 03distributed_data_generator — DDP-aware infinite data loader
- 04torch.compile + Triton warmup — why the first steps take 163ms
- 05Long-Short Sliding Window — Flash Attn 3 + YaRN curriculum
- 06Smear Gate — 1-token lookback inside the embedding
Lineage
From the actual modded-nanogpt speedrun (Keller Jordan repo). Loss timings are illustrative; trace spans are from the real Perfetto run on 8× H100 GPUs. val_loss=3.278, step_avg_ms=67.8.