Lab

Before the program can load weights or allocate memory, it needs to know the model's dimensions. Think of Config as the blueprint before construction — it tells the program how many layers to build, how wide each one is, and how big the vocabulary is. Without it, every allocation and every loop bound would be a guess.

Each field describes a different dimension of the model's shape: - dim — how many numbers represent one position in the model at any layer (1024 here) - hidden_dim — how wide the FFN layer expands to inside each transformer block (3072) - n_layers — how many times the attention+FFN block repeats (28 for Qwen3-0.6B) - n_heads / n_kv_heads — how many parallel attention patterns run simultaneously. Qwen3 uses Grouped Query Attention where 16 query heads share 8 key/value heads, saving memory without losing much quality - vocab_size — how many possible tokens exist (151,936 — every possible subword chunk the model can read or write) - seq_len — the maximum context window: how many tokens can be in the conversation at once - head_dim — how wide each individual attention head is (dim / n_heads)

Nothing in Config is a weight — these are just integers. They're parsed from a header file once at startup. Every buffer size, every matmul dimension, every loop bound in the entire program comes from here.