Quick answer: a genuinely offline-first Flutter AI app runs a small, quantized model on-device for the tasks that must work with no connectivity, and calls a larger cloud LLM for anything that needs deeper reasoning when a connection is available — the architecture decision is the routing logic between the two, not a choice of one over the other.
Why "On-Device vs. Cloud" Is the Wrong Framing
Most mobile AI features fail one of two ways: they assume connectivity that field users (logistics, healthcare, retail, agriculture) don't reliably have, or they try to cram a capable-enough model onto the device and end up with something too weak to be useful. The pattern that actually works treats on-device and cloud inference as two tiers with different jobs — matching how we approach on-device ML more generally in Flutter apps, extended here specifically to LLMs rather than classification models.
The On-Device Runtimes Available in Flutter Today
Two real options exist for running an LLM directly on a phone from a Flutter app:
- Google's MediaPipe LLM Inference API runs quantized models (Gemma and other compatible architectures) directly on-device across Android, iOS, and web, documented in the official MediaPipe LLM Inference guide. It's the more production-supported path today, with an official Flutter integration path through packages built on top of it.
- llama.cpp, the widely used C/C++ inference engine for running quantized GGUF-format models on consumer hardware, can be bound into a Flutter app through community Dart bindings (search pub.dev for current
llama_cpp-family packages before committing — this ecosystem moves fast and package maintenance status matters more than feature lists). It gives you access to a much wider range of open-weight models than MediaPipe's supported set, at the cost of more manual integration work.
Both require quantization — compressing model weights (typically to 4-bit or 8-bit) to fit in a phone's memory and run at usable speed — which trades a small amount of model quality for the ability to run at all on-device.
The Trade-Offs, Concretely
| On-device LLM | Cloud LLM | |
|---|---|---|
| Works with no connectivity | Yes | No |
| Latency | Low, no network round-trip | Depends on network + provider latency |
| Model capability | Limited to what a quantized model on phone hardware can hold | Full-size frontier models |
| Data privacy | Data never leaves the device | Data sent to a third-party API |
| Per-request cost | None after the app ships | Ongoing API cost per request |
| Battery/thermal impact | Real — sustained inference drains battery and can throttle the device | None on-device |
A Routing Pattern That Uses Both
The architecture we build for clients is a router, not a single model choice: the app checks connectivity and task complexity, and picks the tier accordingly. Simple, well-defined tasks — summarizing a note, classifying an intake form, answering from a small local knowledge base — route to the on-device model, working identically with or without a connection. Complex or open-ended requests route to a cloud LLM when connectivity allows, and either queue for later or fall back to a narrower on-device response when it doesn't. This mirrors the same underlying decision covered in our RAG vs. fine-tuning framework — pick the tool based on the actual constraint (here, connectivity and task complexity) rather than defaulting to whichever is more impressive to build.
Flutter's platform channel architecture, which the framework uses to bridge Dart code to native Android/iOS APIs, is what makes this workable in a single codebase — the on-device inference runs through native bindings, while the cloud path is a normal HTTP client call, both orchestrated from the same Dart application logic.
Where This Fits for Mobile Product Teams
If your product has real offline-usage scenarios — field service, healthcare visits in low-connectivity areas, logistics — build the routing decision in from the start rather than retrofitting it after a cloud-only AI feature ships and immediately breaks for a meaningful slice of your users. It's a genuinely different mobile architecture decision, not a copy-paste of a web app's AI integration pattern, and it's the reason offline-first mobile AI is its own specialization rather than "the mobile version of the web app."
If you're building AI functionality into a Flutter app and need it to actually work offline, reach out at info@digit.com.pk.