Language models are discussed either as approaching general intelligence or as autocomplete. Neither description is much use for predicting what they will and won't do.
A more accurate account of the mechanism is more informative than either.
The training objective
The core training process is prediction. Given a sequence of text, predict what comes next. Repeat across an enormous corpus, adjusting parameters to reduce prediction error.
That sounds trivially simple and produces surprising capability, because predicting text well requires modelling a great deal about what the text describes.
To predict the end of a sentence about physics, you need something that behaves like a representation of the physical claim. To predict the conclusion of an argument, something that behaves like the structure of that argument.
Whether that constitutes understanding is a genuinely contested question that the mechanism alone doesn't settle.
The subsequent training
A model trained only on prediction produces text resembling its training data, which isn't the same as being useful.
Further training stages shape behaviour: instruction tuning on examples of following instructions, and preference-based training where the model is optimised against human judgements about which responses are better.
This is where a great deal of the observable character comes from — the tendency to be helpful, to hedge, to structure responses, to decline certain requests.
It also introduces a specific failure mode. Optimising against human preference ratings selects for responses humans rate highly, and humans rate confident, agreeable, well-structured responses highly regardless of accuracy.
Which is a partial explanation for confident wrongness and for a tendency towards agreement.
Why they produce false statements
The most consequential failure mode and the one most misunderstood.
A model generates text likely to follow the preceding text. It has no separate representation of what is true, and no mechanism for checking a claim against a source.
A plausible-sounding false statement and a plausible-sounding true statement are, from the generation mechanism's perspective, similar objects.
Which means this isn't a defect to be patched. It's a consequence of how generation works, and it's why the problem persists across model generations even as accuracy improves.
The characteristic pattern: fabricated citations, invented details, and plausible specifics that don't exist. These appear precisely where the model has learned the shape of an answer without having reliable content for it.
Where they're strong
Given the mechanism, the pattern of capability is fairly predictable.
Transformation. Rewriting, summarising, changing format, translating between styles. The source content is provided, so the model isn't generating facts.
Drafting. Producing a first version to be edited, where the value is in structure rather than in accuracy.
Explanation of well-covered topics. Where the training data contains extensive consistent material.
Code. Where output is verifiable by running it, which provides a check the model itself lacks.
Pattern extraction from provided text. Again, source material supplied.
Where they're weak
Anything requiring reliable factual accuracy without verification. Particularly specific details, figures, citations, quotations and dates.
Recent events. Training data has a cutoff, and models frequently don't reliably know what they don't know about the period after it.
Arithmetic and precise reasoning. Improved considerably and still not reliable in the way a calculator is.
Knowing their own limits. A model's expressed confidence correlates imperfectly with accuracy.
Retrieval and tools
The main architectural response to the accuracy problem.
Rather than relying on parameters, the system retrieves relevant documents and provides them as context, so the model is summarising supplied material rather than generating from memory.
This substantially improves factual reliability and doesn't eliminate the problem — models can still misrepresent retrieved content, and retrieval can surface the wrong documents.
Tool use — calling a calculator, running code, querying a database — addresses specific weaknesses by delegating to systems that are reliable at those tasks.
Both are better understood as engineering around a limitation than as removing it.
Practical guidance
Use them for tasks where you can verify the output, or where accuracy isn't the point.
Provide source material rather than relying on recall, where accuracy matters.
Treat every specific factual claim — a figure, a citation, a name, a date — as requiring verification.
Be aware that pushing back frequently produces agreement rather than correction, which means agreement is weak evidence.
And notice when you're being persuaded by fluency. Well-structured confident prose is what these systems produce regardless of whether the content is sound, which decouples two signals that in human writing are correlated.
Context windows and what they mean
A practical property worth understanding, because it determines what these systems can work with.
The context window is the amount of text a model can consider at once — the conversation so far plus any material you have supplied. Everything outside it is unavailable.
Windows have grown enormously, and there is research indicating that performance degrades for material in the middle of very long contexts, with information at the beginning and end better attended to.
The practical implication: supplying a very large document does not guarantee the model has effectively used all of it. Extracting the relevant sections yourself frequently produces better results than supplying everything and hoping.