Code · architecture · study mode
Without normalization, numbers grow out of control as they pass through 28 layers of matrix multiplication — RMSNorm resets the scale after each layer so the math stays stable.
Full explanation below the code →
Imagine multiplying a vector by a matrix 28 times in a row. Even if each multiplication changes the values by a small amount, those changes compound. Numbers drift toward very large or very small values — eventually becoming infinity or zero, making the entire output meaningless. Normalization prevents this by resetting the scale of the activations at key points.
RMSNorm does this in three steps:
1. Compute the root-mean-square of the current values: ss = sum(x[i]²) / dim, then rms = sqrt(ss + eps) — a measure of the overall magnitude
2. Divide every value by that RMS — rescaling everything to roughly unit magnitude
3. Multiply by a learned per-channel weight vector — letting the model control the scale it actually
wants for each dimension
The result: activations enter the next matmul at a predictable scale, regardless of what happened before.
Qwen3 uses RMSNorm instead of the older LayerNorm (which also subtracts the mean). Skipping the mean subtraction makes it faster and slightly simpler — and it turns out mean subtraction isn't necessary for stability.
Without normalization, deep stacks amplify activation magnitude — matmuls explode or vanish.
Why does a 28-layer model need normalization between layers if each individual matmul seems fine?