Lab

A hash table bakes token-pair knowledge into every embedding lookup

Standard transformers look up one embedding per token. But language is full of bigram patterns — "New York", "don't", "machine learning" — where the meaning of a token depends strongly on the token before it. The model can learn these eventually through attention, but it costs depth and compute.

PR #201 adds a second embedding table indexed by hashed (previous token, current token) pairs. At every position, the model gets both the token embedding and a bigram signal for free, before the first attention layer. The hash maps 2.5 billion possible bigram pairs into 251,519 buckets using XOR of two large multipliers. Collisions are handled by a sign trick that gives each pair a near-orthogonal direction even from the same bucket.