Code · architecture · study mode
For each token the model builds three vectors — a Query (what am I looking for?), and Keys/Values (what does the past contain, and what should it pass on?).
Full explanation below the code →
Attention lets a token look back at every earlier token and decide *how much to care* about each one. For the token being processed right now, the model has already produced three projections (the Q/K/V matmuls from the previous stage):
- Query (Q) — "what am I looking for?" — the vector for the current token (line 365). - Key (K) — "what do I contain?" — one per past token, stored in the KV cache. - Value (V) — "what information do I pass on?" — also one per past token, in the cache.
The intuition (from the attention primer): treat the query as a search, score it against every key by similarity, softmax those scores into weights, then take the weighted average of the values. A query that matches a key strongly pulls in that key's value.
In the code, line 363 loops over the 16 heads (each head attends independently), line 365 grabs this head's query vector, and line 367 points at this head's row of the attention score buffer att. Everything that follows fills and uses that row.
"Query" here means the attention Q vector for a token position — not a user's question. Each token position produces exactly one query; one token being decoded ⇒ one query vector, which is precisely why decode attention is vector-shaped.
During decode of a single new token, how many query vectors exist for one head — and how many keys/values does that query attend over?