A token is the unit of text an LLM actually reads and writes. Models don’t see letters or whole words — a tokenizer first splits text into chunks from a fixed vocabulary: common words become one token, rarer words split into several, and punctuation usually stands alone. In English, one token averages about ¾ of a word (~4 characters), so 1,000 tokens is roughly 750 words.
Why tokens are the unit that matters
Two things are measured in tokens, and both cost you money:
- API pricing is per million tokens, with input (your prompt) and output (the model’s reply) billed at different rates — output typically costs 3–5× more. Claude Sonnet 5, for example, charges $2 per million input tokens and $10 per million output at its introductory rate.
- The context window — how much the model can consider at once — is a token budget, not a word budget.
The catches worth knowing
Tokenization is language- and content-dependent: code, non-English text (Chinese runs roughly 1–2 tokens per character) and unusual formatting all consume more tokens per unit of meaning. And tokenizers differ between models — the same text can count 10–35% more tokens on one model than another, which quietly changes cost comparisons. Anthropic’s Sonnet 5 tokenizer change, which maps identical text to 1.0–1.35× more tokens than its predecessor, is a live example: the per-token price stayed flat while per-task economics shifted.
Count yours
Paste real text into our Token Calculator to see token counts and estimated API costs across GPT, Claude, Gemini, DeepSeek and more — it runs in your browser, nothing is uploaded.