How Language Becomes Tokens
A visual guide to how AI turns language into pieces it can process and predict.
AI models don't read words
"Build me an app that handles all of my customer issues. Make no mistakes."
When you ask AI to solve all of your problems, it is not actually "reasoning" or understanding what you are saying. Sure, we can start a philosophical debate about what reasoning is under the hood, but what matters is that there are algorithms underneath all of this. These algorithms generate a response by predicting what comes next, one token at a time.
Tokenization is the translation layer between human language and models. But what exactly is a token?
What tokens are and why they matter
A token is a piece of text that a model can process. It can be a letter, a word, part of a word, characters with a space, or even an exclamation mark! Each token gets an ID assigned, which is simply its number in the tokenizer's catalog. The model then uses that ID to look up an embedding. Algorithms can then leverage the embeddings to propose the next most likely token. I will explore the actual algorithms, what embeddings mean, and how production-quality token catalogs are generated in depth later in this series. For now, what matters is that computers really only know how to do math. Token IDs give the computers a compact, consistent way to represent pieces of language for efficient processing.
Some basic definitions before we proceed:
- Algorithm - a repeatable set of steps, often mathematical computations done by a computer.
- Catalog - this is the list of known tokens matching a text representation to an ID. A catalog is one of the inputs into an LLM. Also known as a vocabulary or dictionary.
- Embeddings - these are a list of traits that describe how a token is being used in a sentence. You can picture those traits as coordinates on a giant multidimensional map. Tokens used in similar ways tend to sit closer together, while unrelated uses sit farther apart. The model uses those relationships, along with the rest of the sentence, to help predict what comes next.
- Tokenizer - the specific algorithm by which a piece of text is broken down into smaller pieces and assigned an ID.
The catalog is built ahead of time by looking for useful, reusable patterns across a much larger collection of text. The tokenizer applies that existing catalog to your prompt; it does not create new entries while you type.
Tokens are the interface between human language and the model. A tokenizer converts the prompt into token IDs before processing and converts generated IDs back into text afterward.
Tokens also matter because they are the economic unit by which we pay for AI. Providers usually price input and output by the number of tokens. That count also affects how much text a model can consider at once and how long the model takes to process it.
Why not use whole words or individual characters?
At first, using entire words or sentences sounds more efficient. The prompt at the beginning of the article could use 16 word/space tokens, one sentence token, or 73 individual character tokens. But adding a token to the catalog is not free. The model has to learn and store information about how each token is used and how it relates to the language around it. As the catalog grows, the model needs more memory to store what it learned and more computation to choose the next token.
One request, four ways to divide it
· represents a space.
Sentences make this problem explode. Changing one word, name, punctuation mark, or verb tense creates a different sentence. There is no practical way to include every sentence someone might ever write in a fixed catalog. Most sentences would also appear too rarely during training for the model to learn much from them.
Whole words create the same problem at a smaller scale. Handle, handles, handled, and handling would each need their own entry. Names, product IDs, new terms, spelling mistakes, and every supported language would add even more.
Characters solve the catalog problem because a small set of symbols can represent almost anything. The tradeoff is sequence length. The model now has to process 73 pieces instead of 16, using more computation and more of its limited context. Characters also present a very narrow view into language. From a word or words you can often infer meaning. This is a lot harder to do from a single character.
Subwords balance these two extremes. They keep the catalog manageable while representing language with fewer pieces than individual characters. So tokenization becomes a balancing act.
Characters
73 tokens
~100 entries
Subwords
20 tokens
~50K entries
Words
16 tokens
~1M entries
Sentence
1 token
Unbounded
The catalog sizes and limits shown here are illustrative; the exact numbers depend on the languages supported, how the model is built, and the available computing resources.
How a sentence becomes token IDs
Now that we have established why a catalog needs reusable pieces, we can explore how a prompt becomes token IDs. This step becomes a mechanical and repeatable process. When the prompt is received, it is first passed through the tokenizer. At this stage it is broken down into subwords, the subwords are looked up in the catalog, and the IDs are returned in the proper order. Keep in mind that token ordering and spaces as part of the token are critical. You can think of this as a normalization step: the prompt is converted into a consistent, ordered sequence of catalog IDs before it is processed by the model.
From text to token IDs
One sentence, then pieces, then IDs
Showing stage 1: Read the sentence
Build me an app that handles all of my customer issues. Make no mistakes.
As you can see, the result is an ordered list of catalog references. The IDs themselves are arbitrary. ID 311 is not more important or meaningful than ID 17; it simply points to a different catalog entry. Those IDs are only useful to the model paired with that tokenizer. When the model later generates token IDs, the tokenizer uses the same catalog to convert them back into text. Change the tokenizer or the catalog and the IDs hold no meaning.
A note about languages
These models are trained on very large amounts of data from the internet. The training material determines how the catalog gets built. Therefore, if a language such as English appears more often than others, the model will operate on that language more efficiently. However, let's say you have a model that is trained only on data in English. That model will still be able to represent Spanish text. Both languages share similar character sets, but the model may try to apply English language patterns to Spanish prompts and outputs.
Tokenization only asks if a model can represent the set of characters. Training data is what allows a model to derive linguistic patterns and meanings.
Why typos usually do not break the model
A misspelled word does not need its own entry in the token catalog. Consider recieved, a common misspelling of received.
The tokenizer may divide the two spellings differently, but it can still represent both using smaller known fragments. If no useful
subword is available, many modern tokenizers can fall back to individual bytes or characters. The typo may take more tokens, but it
does not become an unreadable "unknown word."
Representing the typo is only the first half of the answer. The model also considers the tokens around it. In a sentence such as
I recieved the package yesterday, words like package and yesterday make the intended meaning much less ambiguous. Models are
also trained on large collections of real text containing spelling mistakes, abbreviations, and informal language, so many common
errors are patterns they have encountered before.
This is why a model can often respond as though the word had been spelled correctly, and may even produce the corrected spelling in its answer. Tokenization ensures the input can be represented; training and context help recover the likely intent. That recovery is not guaranteed. Typos in names, identifiers, numbers, or very short prompts provide less context and can still change the result.
Why tokenization choices matter
Tokenization is the first step of many. It is also the cornerstone of the AI economy. Tokenomics, if you will.
The choices made here compound across cost, speed, context, and performance. In the rest of this series I will explore token economics, embeddings, tokenization algorithms, how catalogs are built, and more. There is a lot of math hiding underneath the magic. Unfortunately for me, I plan to explain it.