A language model never sees letters. It sees tokens, which are fragments of text assigned numeric identifiers, and that intermediate layer explains a surprising share of model failures.
What a token actually is
Tokenizers are built by scanning a large text sample and finding the fragments that recur most often. Common words become single tokens while rare ones are split into pieces.
The vocabulary is fixed before training begins and cannot change afterwards. Everything the model ever reads or writes must be expressed in that fixed set of fragments.
This is a compression decision rather than a linguistic one. The tokenizer is optimising for compact representation of the sample it was built from, not for meaning.
Spelling and arithmetic suffer directly
Asking a model to count letters in a word requires it to reason about characters it never received individually. The word arrived as one or two opaque fragments.
Numbers behave the same way. Depending on the tokenizer, a long number may split at arbitrary points, so digits that should align by place value do not.
Models can still learn these tasks, but they are learning to reconstruct information the input format discarded. That is why performance is inconsistent rather than uniformly absent.
Languages are not treated equally
A tokenizer built mostly on English text represents English efficiently and everything else poorly. The same sentence in another script can consume several times as many tokens.
That inefficiency shows up as cost, since billing and context limits are measured in tokens. Users writing in some languages pay more for the same content.
It also reduces effective context. A fixed window holds noticeably less meaning when each word costs more fragments to express.
Tokenization sets the economics
Every part of the serving pipeline is priced per token, so the tokenizer determines throughput as much as the model architecture does.
Improving compression by even a small margin lowers cost across every request. This is why tokenizer design receives attention out of proportion to its apparent simplicity.
Why the fix is not straightforward
Working directly on characters or bytes removes these problems but lengthens every sequence substantially, which raises compute cost for the same amount of text.
Alternative schemes that group bytes adaptively are being explored, trading a more complex model for a simpler input representation.
Until one of those approaches becomes standard, tokenization remains a quiet source of behaviour that looks like reasoning failure but is really a representation artefact.