// block: bm_sizes — asymmetric sliding windows · lines 1222–1232$ study train_gpt.py --block bm-sizes11 layers, 3 window types: short (8 layers), long (L3 + L10 — wider context bridges), full (L6 — no … ✓
Explanation
11 layers, 3 window types: short (8 layers), long (L3 + L10 — wider context bridges), full (L6 — no mask). ws_schedule=(3,7,11): windows grow during training. YaRN rescales RoPE when ws_long changes.
Think about
Why use full attention only at L6 rather than at L0 or L10?
// architecture
Live diagram
100%
05 · train_gpt.py line 1222 — bm_sizes, YaRN
Flash Attn 3 + Sliding Window
Two context windows per layer — and how they grow during training
Sliding-window attention limits each token to a fixed context window. The speedrun uses this for 9 of 11 layers, but asymmetrically: a short window for most layers and a longer window for layers 3 and 10 (wider context bridges). Layer 6 uses full attention.
The window sizes grow during training on a curriculum schedule — ws_schedule=(3,7,11). This is why YaRN is needed: when the window grows, the model encounters position indices further apart than anything it trained on. YaRN rescales RoPE angular frequencies smoothly.
Most predictions are highly local — the last few hundred tokens contain the needed signal. Short-window layers handle this efficiently. Layers 3 and 10 act as wider context bridges. L6's full attention is the single fully-expressive layer in the stack middle.
YaRN — why RoPE needs rescaling
When the window grows, the model encounters token pairs at distances it was never trained on. YaRN rescales RoPE angular frequencies: long-period components are left unchanged; short-period ones are scaled down. Context length extension without catastrophic forgetting.