Code · architecture · study mode
The model never sees words or letters — only integers. The vocabulary is the lookup table that maps every possible token (a text chunk, usually 3-4 characters) to a unique integer ID and back.
Full explanation below the code →
What is load_vocab()? When you type "Hello" to the model, it doesn't process letters H, e, l, l, o. It processes token IDs — integers. A token is a chunk of text that the model treats as an indivisible unit. "Hello" might be a single token (ID: 9906) or split across two tokens depending on context. The vocabulary is the complete list of all 151,936 possible tokens, each assigned a permanent integer ID.
Why does this exist? The model only does math — matrix multiplications, additions, softmax. It can't multiply a letter. Token IDs are the bridge: each integer maps to a row in the embedding table (a vector of 1024 floats) that the model can actually process.
What load_vocab() does:
It reads vocab.txt, where each line is one token string. Line N becomes token ID N. The reverse is trivial: vocab[token_id] returns the string. The forward direction (string → ID) is more complex and happens during encode().
What's in Qwen3's vocabulary:
- Regular subword tokens (common word pieces like Ġhello, world)
- Special control tokens like <|im_start|> and `` that structure the conversation
- Byte-level fallback tokens for rare characters
- Characters from many languages
The critical constraint:
vocab_size in Config must exactly match the number of lines in vocab.txt. A mismatch means the embedding table (which has vocab_size rows) and the logits vector (which has vocab_size scores) are sized differently from what the tokenizer produces — silent corruption.
Token IDs are the integer bridge between UTF-8 text and floating-point embeddings.
Why does the model use token IDs (integers) instead of processing characters or words directly?