Smear Lab

running

◷train_gpt.py▸smear_gate definition◎learner＄no GPU

Blocks

Quick summary

nn.Linear(12, 1, bias=False) — 12 weights total. Reads only the first 12 dims of a 768-dim embedding to produce a gate…

Full explanation below the code →

fba-lab — train_gpt.py · smear_gate definitionexecuting

// block: smear_gate definition · lines 1079–1082$ study train_gpt.py --block smear-gate-initnn.Linear(12, 1, bias=False) — 12 weights total. Reads only the first 12 dims of a 768-dim embedding… ✓

Explanation

nn.Linear(12, 1, bias=False) — 12 weights total. Reads only the first 12 dims of a 768-dim embedding to produce a gate scalar. init zeros → model starts with no smearing, learns to blend if useful.

Think about

Why read only the first 12 dimensions instead of the full 768?

// architecture

Live diagram

06 · train_gpt.py line 1239 · @classiclarryd

Smear Gate

1-token lookback: blending neighboring embeddings before any transformer layer sees them

The bigram hash injects a "what are the two tokens?" signal by looking up a pre-computed embedding table. The smear gate does something complementary: it takes the actual embedding vector of the previous position and blends a learned fraction of it into the current position's embedding — before the transformer stack, before normalization, before attention.

The gate itself is tiny: nn.Linear(12, 1, bias=False) — just 12 trainable weights. It reads only the first 12 dimensions of the current position's token embedding and produces a single scalar in (0, 1) via sigmoid. That scalar is scaled by a global learned value (smear_lambda, initialized to zero).

# self.smear_gate = nn.Linear(12, 1, bias=False) — 12 weights total
# nn.init.zeros_(self.smear_gate.weight) — starts disabled
# smear_lambda = self.scalars[4*num_layers] — learned global scalar, init 0

# train_gpt.py line 1239-1240
smear_gate_out = smear_lambda * torch.sigmoid(
self.smear_gate(x[1:, :self.smear_gate.weight.size(-1)])
) # shape (T-1, 1) — one gate scalar per position
x = torch.cat([x[:1], x[1:] + smear_gate_out * x[:-1]])
# position 0: unchanged (no previous position)
# positions 1..T: x_t ← x_t + gate_t × x_{t-1}
# Applied BEFORE norm(x[None]) and before the transformer layers

Bigram hash vs smear gate

Both inject left-context before the transformer stack, but at different levels of representation:

Bigram: discrete — hash of token IDs → lookup table → stored vector
Smear: continuous — embedding of prev token × per-position scalar gate
Both initialized to zero — model can ignore either if not useful
Both applied before norm and transformer stack

Why only 12 input dimensions?

The gate doesn't need to understand the full meaning of the current token — it just needs a cheap signal for "how much of the previous token should carry forward?" The first 12 dims of a well-trained token embedding encode substantial token class information. 12×1 = 12 parameters — effectively zero overhead.

lr_mul=0.01 in optimizer — trained 100× slower than main weights
Contribution: @classiclarryd — same contributor as the original bigram hash

1-token lookback: blending neighboring embeddings before any transformer layer sees them

Bigram hash vs smear gate

Both inject left-context before the transformer stack, but at different levels of representation:

Bigram: discrete — hash of token IDs → lookup table → stored vector
Smear: continuous — embedding of prev token × per-position scalar gate
Both initialized to zero — model can ignore either if not useful
Both applied before norm and transformer stack

Why only 12 input dimensions?

lr_mul=0.01 in optimizer — trained 100× slower than main weights
Contribution: @classiclarryd — same contributor as the original bigram hash