Code · architecture · study mode
softcap(x, cap=30) = tanh(x/30) * 30 — bounds extreme logits smoothly.
Full explanation below the code →
softcap(x, cap=30) = tanh(x/30) * 30 — bounds extreme logits smoothly.
What happens to a logit of 100 after softcap?
🎬 Video lesson coming soon
This step's video hasn't been recorded yet. The text lesson and Rune's deep-dives cover the same ground in the meantime.