Speedrun — #3 Longer Context

running

◷train_gpt2.py▸sequence_length 32768◎learner⌁06-longer-sequence＄no GPU

baseline

GPT-2 d12

Starting architecture and training loop.

step 01

optimize

Speedrun step

Architecture diffs and optimizer changes.

steps 02–05

active

target

val loss 3.28

Cumulative loss timeline and finale.

step 06

Blocks

Quick summary

Context window jumps from 1024 to 32768 — 32× more tokens per forward.

Full explanation below the code →

fba-lab — train_gpt2.py · sequence_length 32768executing

// block: sequence_length 32768 · lines 10–20$ study train_gpt2.py --block seq_lengthContext window jumps from 1024 to 32768 — 32× more tokens per forward. ✓

Explanation

Context window jumps from 1024 to 32768 — 32× more tokens per forward.

Think about

How does memory scale with sequence length for attention?

// architecture

Live diagram

What changed vs 05-softcap

+ from contextlib import contextmanager

What it bought

Learn this step, your way

WatchReadDeep Dive

🎬 Video lesson coming soon

This step's video hasn't been recorded yet. The text lesson and Rune's deep-dives cover the same ground in the meantime.

Lab

Speedrun — #3 Longer Context

Live diagram

What changed vs 05-softcap

sequence_length 32768

Updated T bounds assert

What it bought

Learn this step, your way

Speedrun — #3 Longer Context

Live diagram

What changed vs 05-softcap

sequence_length 32768

Updated T bounds assert

What it bought

Learn this step, your way