On-Device ML in Flutter: TFLite to CoreML without the Pain

On-device ML unlocks two things most server-side inference cannot: offline capability and user privacy. For the healthcare and fintech apps we build at DIGIT, both matter. Here is how we approach cross-platform on-device ML in Flutter.

The Core Problem

Flutter runs on both iOS and Android, but the ML runtimes are different: TensorFlow Lite on Android, CoreML on iOS. Writing platform-specific ML code defeats the purpose of cross-platform development. Our solution is a thin abstraction layer that exposes a single Dart interface regardless of the underlying runtime.

The Abstraction Layer

We define a MLInferenceEngine abstract class in Dart with three methods: load(modelPath), run(input), and dispose(). Platform implementations live behind Flutter's platform channel mechanism. On Android, the implementation wraps TFLite. On iOS, it wraps CoreML via a Swift wrapper.

The key insight: both runtimes take a tensor as input and return a tensor as output. Normalising the input/output format at the Dart layer means the Flutter business logic never needs to know which runtime it is talking to.

Model Conversion

We train models in PyTorch or TensorFlow, then convert: PyTorch → ONNX → TFLite (Android), PyTorch → CoreML Tools (iOS). The conversion step is where most teams run into trouble. CoreML Tools has strict requirements on operator support — anything using dynamic shapes or custom ops will fail silently or produce wrong outputs.

Our practice: validate the converted model's outputs against the training framework outputs on 100 test samples before shipping. A 0.01 difference in output is expected; a 5% difference indicates a conversion error.

Performance Considerations

• Quantise to INT8 for Android; the accuracy trade-off is minimal for classification tasks and inference is 2–4× faster.
• On iOS, use the Neural Engine (ANE) target for CoreML — check MLComputeUnits.all in your CoreML config. ANE inference is 5–10× faster than CPU on supported hardware.
• Pre-warm the model on app startup; the first inference call is always slow due to model loading.

This pattern has served us well across 8 production mobile apps with on-device ML requirements.