What is a token? The unit every AI limit is measured in

A token is the basic unit that language models read and write, usually representing about three-quarters of an English word. Tokenization splits your text into these chunks before the model processes it.

Overview

Tokens are essential for understanding how AI systems operate and their limitations. They help in managing context windows, API pricing, speed, and even model behavior.

Why Tokens?

• Vocabulary Tradeoff: Using whole words limits the vocabulary but speeds up processing. Subword pieces like BPE (Byte Pair Encoding) balance between a large number of tokens (30k-150k) and maintaining efficiency.
• Character Approximation: On average, 1 token is about 4 characters or 0.75 English words.

Practical Implications

• Context Windows: These are token budgets that limit the amount of text a model can process at once (e.g., "128K context" means your input and output must fit within 128,000 tokens).
• API Pricing: Costs are often calculated per token, both for input and output.
• Speed: Tokenization speed is measured in tokens per second (tok/s), which can be converted to words per second.

Examples

• Page of Text: A typical page contains about 500-700 tokens.
• Code vs. English: Code tokenizes differently, often resulting in more tokens for the same meaning due to symbols and indentation.
• Non-English Languages: These can have significantly higher token counts per word compared to English.

Weird Model Behavior

• Tokenization Issues: Classic tasks like counting strawberries or performing exact string matching may behave strangely because models operate on tokens, not individual characters.

Try It Yourself

You can experiment with tokenizers provided by platforms like OpenAI or use local tools such as Ollama and LLaMA.cpp to see how text is split into tokens. This helps in building a better understanding of the concept.

Concrete examples: "The server is down" is 5 tokens. The same sentence in Spanish — "El servidor está caído" — splits into more pieces, because tokenizers are trained mostly on English text; the same meaning can cost 1.5–3× more tokens in other languages, a real speed and cost penalty. Code behaves similarly: def hello_world(): becomes several tokens for what reads as two words, since symbols and underscores split words apart. Exact counts vary by model, because every model family ships its own tokenizer.