Training a large model is expensive and highly visible. Serving it is less visible and, for any widely used system, eventually the larger line on the bill.
One expense repeats and the other does not
A training run happens once and produces a fixed set of weights. Whatever it cost, that number stops growing the moment the run finishes.
Serving cost scales with usage. Every question asked consumes compute again, so a popular model accumulates spending indefinitely against a training cost that is already settled.
For a product with sustained traffic, the crossover arrives sooner than most planning assumes. After that point all economics are inference economics.
What a single response consumes
Generating text is sequential. Each output token requires a pass through the model, so a long answer costs proportionally more than a short one.
Reading the prompt is cheaper per token because it can be processed in parallel, which is why input and output are priced differently by most providers.
The asymmetry means that verbose outputs are the expensive part of a conversation, and shortening responses saves more than shortening questions.
Context length multiplies everything
Attention cost grows faster than linearly with sequence length, so doubling the context does more than double the work in the naive case.
Caching intermediate state avoids recomputing the prompt on each new token, but that cache occupies memory that could otherwise hold more concurrent users.
Long-context features therefore trade directly against how many people a given cluster can serve at once. The limit is memory as much as arithmetic.
Serving is a capacity problem
Requests arrive unevenly across the day and week, and accelerators cannot be provisioned instantly in response. Capacity must therefore be held for peaks that occur only part of the time, and it sits idle for the rest.
Batching many requests together improves utilisation because the weights are loaded once and applied to several inputs. That adds waiting time for whoever arrives first, so operators trade latency against efficiency continuously rather than choosing a setting once.
Model weights also have to be resident in accelerator memory before anything can be served, which sets a floor on hardware even at low traffic. A large model has a minimum footprint regardless of how few people are using it.
Optimisation targets moved accordingly
Techniques that reduce weight precision, prune redundant computation or distil a large model into a smaller one all target serving rather than training. None of them make the original run cheaper, because that run has already happened.
Routing systems that send easy queries to smaller models follow the same logic, since most requests do not need the largest available system.
The pattern across all of them is the same: accept a modest quality cost in exchange for a repeated saving that never stops accruing.