Code · architecture · study mode
These activation buffers are calloc'd in CPU RAM here; the CUDA port allocates the identical buffers in GPU VRAM with cudaMalloc — with one deliberate exception.
Full explanation below the code →
malloc_run_state allocates every activation buffer the forward pass scribbles on: x, xb, q, k, v, the attention scratch att, and the key_cache/value_cache (lines 84–96). In run.c these are calloc — ordinary CPU heap memory — because the compute runs on the CPU.
In the CUDA port (runcu.cu) the exact same buffers are cudaMalloc'd in GPU VRAM, because the compute runs on the GPU. The rule is: *the data must live where the math runs.* If activations sat in CPU RAM while kernels ran on the GPU, every op would copy data across the PCIe bus and performance would collapse. In short: run.c keeps weights (mmap) and activations (calloc) in CPU RAM; runcu.cu keeps both in GPU VRAM.
The one exception is `s->logits` (line 94). It stays on the CPU even in the GPU port, because the sampler (sample()) runs on the CPU. So the GPU computes logits into a device buffer and copies them back to this host buffer once per token.
Deciding "forward() runs on the GPU" forces a decision about *everything it touches*. Once you commit the activations to VRAM, every step of the layer must also run on the GPU — you can't drop back to a CPU loop mid-layer without paying a PCIe round trip each time.
Why is s->logits allocated on the host even in the CUDA version, when every other buffer moves to the GPU?