Language models improved for years mainly by being fed more text. That approach has hit a supply problem, because the amount of useful human-written material available for training is not unlimited.
The usable supply is smaller than the web
The open web is enormous, but most of it is duplicated, machine-generated, boilerplate or navigational text. Once those are stripped out, the genuinely distinct written material shrinks dramatically.
Training pipelines also deduplicate aggressively, because repeated passages cause models to memorise rather than generalise. Deduplication removes a large fraction of what was collected in the first place.
What remains is a corpus that grows only as fast as people write new things. That growth rate is slow compared with how quickly training runs consumed the existing stock.
Not all text carries the same value
Technical documentation, edited books and careful long-form writing teach a model far more per word than forum chatter. The scarce material is precisely the material that took effort to produce.
This is why data curation became a competitive advantage rather than an afterthought. Two labs with similar compute budgets can end up with noticeably different models purely through selection.
Curation also introduces judgement calls about what counts as quality, and those choices propagate into model behaviour. The filter is part of the training method, not a neutral preprocessing step.
Synthetic text feeds back on itself
An obvious response is to generate training text with models themselves. This works in narrow cases where the output can be checked, such as code that either compiles or does not.
In open-ended writing the risk is that the model learns its own tendencies back. Errors and stylistic quirks get reinforced rather than corrected, and diversity in the corpus narrows.
Filtering synthetic data against a verifier restores some value, but verification is only available where correctness is mechanically checkable. Most prose does not fall into that category.
Licensing changed where data comes from
Large publishers and platforms increasingly restrict automated collection and negotiate access instead. That turns a formerly free input into a contracted one with a price attached.
The practical effect is that data access now depends on commercial relationships as much as engineering. Smaller organisations face a harder path to comparable corpora.
Efficiency replaced volume as the lever
With the raw supply constrained, attention moved to getting more out of each token through better training recipes, curriculum ordering and longer training on curated subsets.
Post-training work also grew in importance, because shaping how a model uses what it already knows does not require new text at the same scale.
The result is that progress now depends less on scraping more and more on making better use of a corpus that is close to fully harvested.