A significant amount of engineering effort is going into running machine learning models on phones and computers rather than on remote servers.
The trade-offs are worth understanding, because they determine which applications work locally and which don't.
Why local matters
Privacy. Data processed on the device doesn't leave it. For anything involving personal content — messages, photos, health data, documents — this is a substantial difference, and it's the strongest argument.
Latency. No round trip to a server. For anything interactive, particularly real-time processing, this is the difference between usable and not.
Availability. Works without connectivity, and without depending on a service remaining operational.
Cost. No per-request charges. For a manufacturer, moving processing to the device shifts cost from their servers to hardware the customer bought.
That last point explains a good deal of the industry investment.
The constraints
Memory. The binding constraint. A model must fit in available memory alongside everything else running.
Large models require substantially more memory than phones have. Which means local models are considerably smaller than server-hosted ones, with correspondingly reduced capability.
Power. Running a model consumes energy and generates heat. Sustained processing on a phone drains the battery and triggers thermal management.
Processing capability. Dedicated neural processing units in modern devices are efficient at these workloads and are still far below what server hardware provides.
Storage. Models occupy space, and users are already constrained.
How models are made to fit
Several techniques, each trading capability for size.
Quantisation. Storing parameters at lower numerical precision. Reduces memory substantially with some accuracy cost, and modern techniques have made the cost surprisingly small.
This is the most important technique in practice.
Distillation. Training a smaller model to reproduce a larger one's behaviour. Produces a compact model that performs better than one trained directly at that size.
Pruning. Removing parameters that contribute little.
Task-specific models. A small model trained for one narrow task can perform well at it while being far smaller than a general model.
This is why on-device features tend to be specific — transcription, translation, image processing — rather than general capability.
What runs locally now
Speech recognition, which has been on-device for some time and works well.
Image processing, including the computational photography pipeline.
Translation for common language pairs.
Text prediction and correction.
Content classification and search within photos.
Smaller general language models, which are genuinely useful for constrained tasks and clearly less capable than server-hosted alternatives.
The hybrid approach
The architecture most systems are converging on.
Simple requests handled locally. Complex ones routed to a server. The routing decision made automatically, sometimes without the user knowing which happened.
This gets most of the latency and privacy benefit for common cases while retaining capability for hard ones.
The complication is that the privacy benefit depends on the routing, and users generally can't see which requests went where. Several implementations have introduced ways to indicate or control this, with varying transparency.
Some providers have introduced server-side processing with cryptographic guarantees intended to prevent even the provider from accessing the data. Whether such arrangements deliver what they claim is something the security community will assess over time.
Practical implications
For users choosing devices, local processing capability is becoming a genuine differentiator — the amount of memory in particular, since it determines what can run.
For anyone concerned about privacy, features that process locally are meaningfully different from those that don't, and it's worth establishing which is which.
And for developers, the constraint is memory before anything else. A model that fits is usable; one that doesn't isn't, regardless of how good it is.
Where it goes
The trajectory points towards more capability locally, driven by better compression techniques, more memory in devices, and dedicated hardware.
The gap with server-hosted models will persist, because the same techniques improve those too and the resource difference is enormous.
Which suggests a durable division: local for private, latency-sensitive and frequent tasks; remote for anything requiring maximum capability. That's a reasonable arrangement, and the thing worth watching is whether users retain any visibility into which is happening.
The environmental argument
Occasionally offered and worth examining carefully, because it cuts both ways.
Running a model on a device avoids data centre energy consumption for that request, which is the basis of the claim.
Against that: the specialised hardware required is manufactured, and manufacturing carries a substantial footprint. If on-device capability drives faster device replacement, the embodied emissions may exceed the operational saving.
The honest position is that it depends on utilisation and on replacement behaviour, and that neither is well measured.
What is clearer is that efficiency improvements in the models themselves — smaller models achieving comparable results — reduce consumption wherever they run, which is the less contested route.
A practical note for anyone assessing devices on this basis: memory is the specification that determines what can run, and it is frequently not prominent in marketing.
Two devices with identical processors and different memory will differ substantially in which local features work, and that difference will grow as models get larger. For a device you intend to keep for several years, it is arguably the specification most worth paying attention to.