Terminology· 2 min
What are tokens?
Tokens are the chunks of text LLMs actually see — usually 3-4 characters or half a word each.
Key idea
Text is split by a tokenizer (BPE, SentencePiece) before the model touches it. Pricing, context limits and speed are all measured in tokens.
Example
'ChatGPT is amazing!' ≈ 6 tokens. English averages ~0.75 words per token; code and non-English languages are far less efficient.
What to remember
- Non-English text often costs 2-4× more tokens
- Emoji and code use many tokens per character
- Use OpenAI's tiktoken or model-specific tokenizer to estimate cost