Tokenizer
The fixed table that chops text into tokens. Decided before training and never changed afterwards.
Also written: tokenisation, tokenization, tokenising.
Before a model is trained, someone runs an algorithm over a huge pile of text to find the few tens of thousands of fragments that represent it most compactly. That table is the tokenizer, and it is frozen for the life of the model.
This is why token counts are not portable: the same sentence can be 18 tokens to one vendor’s model and 23 to another’s. It is also why you cannot reliably count tokens by counting characters and dividing. An estimate is fine for planning; for a bill, the only authority is the vendor’s own count.