1-token lookback: blending neighboring embeddings before any transformer layer sees them

The bigram hash injects a "what are the two tokens?" signal by looking up a pre-computed embedding table. The smear gate does something complementary: it takes the actual embedding vector of the previous position and blends a learned fraction of it into the current position's embedding — before the transformer stack, before normalization, before attention.

The gate itself is tiny: nn.Linear(12, 1, bias=False) — just 12 trainable weights. It reads only the first 12 dimensions of the current position's token embedding and produces a single scalar in (0, 1) via sigmoid. That scalar is scaled by a global learned value (smear_lambda, initialized to zero).

# self.smear_gate = nn.Linear(12, 1, bias=False) — 12 weights total
# nn.init.zeros_(self.smear_gate.weight) — starts disabled
# smear_lambda = self.scalars[4*num_layers] — learned global scalar, init 0

# train_gpt.py line 1239-1240
smear_gate_out = smear_lambda * torch.sigmoid(
self.smear_gate(x[1:, :self.smear_gate.weight.size(-1)])
) # shape (T-1, 1) — one gate scalar per position
x = torch.cat([x[:1], x[1:] + smear_gate_out * x[:-1]])
# position 0: unchanged (no previous position)
# positions 1..T: x_t ← x_t + gate_t × x_{t-1}
# Applied BEFORE norm(x[None]) and before the transformer layers

Bigram hash vs smear gate

Both inject left-context before the transformer stack, but at different levels of representation:

Bigram: discrete — hash of token IDs → lookup table → stored vector
Smear: continuous — embedding of prev token × per-position scalar gate
Both initialized to zero — model can ignore either if not useful
Both applied before norm and transformer stack

Why only 12 input dimensions?

The gate doesn't need to understand the full meaning of the current token — it just needs a cheap signal for "how much of the previous token should carry forward?" The first 12 dims of a well-trained token embedding encode substantial token class information. 12×1 = 12 parameters — effectively zero overhead.

lr_mul=0.01 in optimizer — trained 100× slower than main weights
Contribution: @classiclarryd — same contributor as the original bigram hash