Tokenizer
Also: tokens, tokenisation
The component that breaks text into tokens, the sub-word units a model actually reads and generates. Token counts, not word counts, drive context limits and API pricing.
Models do not read characters or words directly. A tokenizer first breaks text into tokens, sub-word pieces where a common word may be one token and a rare one several. A rough rule of thumb for English is that a token is about three-quarters of a word.
Tokens are the unit everything is measured in: a model’s context window is a number of tokens, generation speed is quoted in tokens per second, and hosted APIs bill per million tokens. Knowing that helps read specifications and estimate costs, which is why the cost calculator works in tokens.