Code · architecture · study mode
After forward() produces 151,936 raw scores, the sampler decides how to pick one — temperature controls how random the choice is, from always-pick-the-best to weighted random selection.
Full explanation below the code →
What is build_sampler()?
forward() gives us 151,936 scores — one per possible next token. But how do we turn those scores into a single choice? That's the sampler's job. The choice of sampling strategy has a huge effect on the model's behavior.
Temperature in plain English: Temperature (named after thermodynamics) controls randomness. At temperature 0, always pick the highest-scoring token — deterministic, good for coding or factual questions. At temperature 1.0, sample proportionally to the probability each token was assigned — some randomness, natural-feeling text. At temperature 2.0, more adventurous choices, sometimes creative, sometimes incoherent.
Top-p (nucleus sampling):
top-p adds another filter: only consider tokens whose cumulative probability reaches topp (e.g., 0.9). This cuts off the long tail of very unlikely tokens — preventing occasional random nonsense without eliminating all creativity.
Why allocate scratch buffers here?
The sampler is initialized once and reused for every token. Its scratch buffers (probabilities array, sorting workspace) are allocated here, not inside sample() — avoiding per-token allocation overhead.
Same forward() can drive greedy CLI, creative writing, or reproducible benchmarks — policy lives here.
In plain English, what does temperature actually control about the model's word choices?