Speedrun — #2.4 Logit Soft-capping

running

◷train_gpt2.py▸softcap()◎learner⌁05-softcap＄no GPU

baseline

GPT-2 d12

Starting architecture and training loop.

step 01

active

optimize

Speedrun step

Architecture diffs and optimizer changes.

steps 02–05

target

val loss 3.28

Cumulative loss timeline and finale.

step 06

Blocks

Quick summary

softcap(x, cap=30) = tanh(x/30) * 30 — bounds extreme logits smoothly.

Full explanation below the code →

fba-lab — train_gpt2.py · softcap()executing

// block: softcap() · lines 30–35$ study train_gpt2.py --block softcap_fnsoftcap(x, cap=30) = tanh(x/30) * 30 — bounds extreme logits smoothly. ✓

Explanation

softcap(x, cap=30) = tanh(x/30) * 30 — bounds extreme logits smoothly.

Think about

What happens to a logit of 100 after softcap?

// architecture

Live diagram

What changed vs 04-data-loading

Select a change to view diff hunks.

What it bought

Learn this step, your way

WatchReadDeep Dive

🎬 Video lesson coming soon

This step's video hasn't been published on YouTube yet. The text lesson and Deep Dives cover the same ground in the meantime.

Lab

Speedrun — #2.4 Logit Soft-capping

Live diagram

What changed vs 04-data-loading

softcap helper

Softcap in GPT forward

What it bought

Learn this step, your way

Speedrun — #2.4 Logit Soft-capping

Live diagram

What changed vs 04-data-loading

softcap helper

Softcap in GPT forward

What it bought

Learn this step, your way