Why big LLMs don't need these tricks
These three functions — bigram hash, polar express, BOS-aligned generator — are engineering shortcuts for a constrained setting. They help a tiny model hit good loss fast. They are not wrong. They are pragmatic. Production LLMs take a different approach.
+ bigram_embed(hash(t-1, t))
→ attention learns everything else
- 11 layers, 768-dim — limited capacity
- Bigram shortcut pre-injects local pair context before attention
- polar_express orthogonalizes gradients — NorMuon variant
- BOS alignment needed to avoid sequence midpoint cuts hurting loss
- Goal: minimize val loss per GPU-second, not clean architecture
→ causal attention learns context
→ MLP learns higher-order patterns
- 28–96+ layers, 1024–7168 dim — sufficient capacity for everything
- No bigram table — attention learns "New→York" naturally from data
- No polar_express — AdamW or standard Adam with weight decay
- Packing with document attention masks handles multi-doc batches
- Goal: clean scaling behavior, general capability, not speedrun metrics
The token embedding in any LLM is static per token ID. The previous-token information only enters through causal self-attention — which requires capacity (parameters) and data (training tokens) to learn.
A bigger model like DeepSeek-7B (32 layers, 4096 dim, 2T tokens) has enough capacity to learn bigrams, trigrams, syntax, and reasoning through the standard transformer stack alone. The bigram hack would add table collision noise, more hyperparameters, and architectural complexity with no benefit. The bigram trick is not wrong — it is an engineering shortcut that improves validation loss per unit compute in a constrained setting.