Step 3 of 28Skill: Model Training Fundamentals

TRAINING SIMULATION

Speedrun — #2.1 Architecture Tweaks

running

◷train_gpt2.py▸Hyperparameters dataclass◎learner⌁02-architectural＄no GPU

baseline

GPT-2 d12

Starting architecture and training loop.

step 01

active

optimize

Speedrun step

Architecture diffs and optimizer changes.

steps 02–05

target

val loss 3.28

Cumulative loss timeline and finale.

step 06

Blocks

Quick summary

Replaced argparse with a dataclass — single source of truth for hparams.

Full explanation below the code →

fba-lab — train_gpt2.py · Hyperparameters dataclassexecuting

// block: Hyperparameters dataclass · lines 1–30$ study train_gpt2.py --block hyperparametersReplaced argparse with a dataclass — single source of truth for hparams. ✓

Explanation

Replaced argparse with a dataclass — single source of truth for hparams.

Think about

What three architecture-related defaults changed vs step 01?

// architecture

Live diagram

What changed vs 01-baseline

+         self.transformer = nn.ModuleDict(
+             dict(
+                 wte=nn.Embedding(config.vocab_size, config.n_embd),
+                 h=nn.ModuleList([Block(config) for _ in range(config.n_layer)]),
+             )
+         )
+         self.transformer.wte.weight = self.lm_head.weight  # https://paperswithcode.com/method/weight-tying
+         x = self.transformer.wte(idx)  # token embeddings of shape (b, t, n_embd)
+         x = F.rms_norm(x, (x.size(-1),))
+             logits = logits.float()  # use tf32/fp32 for logits
-             logits = self.lm_head(x[:, [-1], :]) # note: using list [-1] to preserve the time dim
+             logits = self.lm_head(x[:, [-1], :])  # note: using list [-1] to preserve the time dim
+             logits = logits.float()  # use tf32/fp32 for logits
-     def configure_optimizers(self, weight_decay, learning_rate, betas, device_type):
-     with open(filename, "rb") as f:
-         header = np.frombuffer(f.read(256*4), dtype=np.int32)
-         print("ERROR: magic number mismatch in the data .bin file!")
-         print("---> HINT: Are you passing in a correct file with --input_bin?")
-         print("---> HINT: Dataset encoding changed recently, re-run data prepro or refer again to README")

What it bought

Learn this step, your way

WatchReadDeep Dive

🎬 Video lesson coming soon

This step's video hasn't been recorded yet. The text lesson and Rune's deep-dives cover the same ground in the meantime.

Lab

Speedrun — #2.1 Architecture Tweaks

Live diagram

What changed vs 01-baseline

RoPE replaces wpe

ReLU² MLP

Depth-aware attention scale

What it bought

Learn this step, your way

Speedrun — #2.1 Architecture Tweaks

Live diagram

What changed vs 01-baseline

RoPE replaces wpe

ReLU² MLP

Depth-aware attention scale

What it bought

Learn this step, your way