Speedrun — #2.3 Dataloading Tweaks

running

◷train_gpt2.py▸DistributedDataLoader◎learner⌁04-data-loading＄no GPU

baseline

GPT-2 d12

Starting architecture and training loop.

step 01

active

optimize

Speedrun step

Architecture diffs and optimizer changes.

steps 02–05

target

val loss 3.28

Cumulative loss timeline and finale.

step 06

Blocks

Quick summary

Shard iterator with per-rank cursor; yields contiguous (B,T) token windows.

Full explanation below the code →

fba-lab — train_gpt2.py · DistributedDataLoaderexecuting

// block: DistributedDataLoader · lines 293–338$ study train_gpt2.py --block dataloaderShard iterator with per-rank cursor; yields contiguous (B,T) token windows. ✓

Explanation

Shard iterator with per-rank cursor; yields contiguous (B,T) token windows.

Think about

What changed in how B is passed to the train loader vs step 03?

// architecture

Live diagram

What changed vs 03-muon

- f'step:{step + 1}/{args.num_iterations} train_loss:{train_loss.item():.4f} train_time:{approx_time:.0f}ms step_avg:{approx_time / timed_steps:.2f}ms tokens_seen:{tokens_seen:.2e} tokens/sec:{tokens_per_second:.2e}',

What it bought

Learn this step, your way

WatchReadDeep Dive

🎬 Video lesson coming soon

This step's video hasn't been recorded yet. The text lesson and Rune's deep-dives cover the same ground in the meantime.

Lab

Speedrun — #2.3 Dataloading Tweaks

Live diagram

What changed vs 03-muon

Loader batch size tweak

Simplified loader init

What it bought

Learn this step, your way

Speedrun — #2.3 Dataloading Tweaks

Live diagram

What changed vs 03-muon

Loader batch size tweak

Simplified loader init

What it bought

Learn this step, your way