Edge AI
Plan local inference around device memory, latency, energy and quality, including quantization and offline behavior.
Before you start: Training and inference. Reading time is an estimate; allow extra time for practice.
What you’ll learn
- Distinguish edge inference from model training and IoT connectivity.
- Budget model storage, runtime memory, latency and energy separately.
- Explain quantization and why the converted model needs evaluation.
- Design offline behavior and a safe model-update process.
On this page
Bring inference close to the input
Edge AI runs model computation near where data is produced or used: on a phone, laptop, embedded device or local gateway. It does not necessarily mean a tiny model, and it does not require the device to be part of an IoT fleet. This guide uses a fictional keyword detector that recognizes a short command locally.
A typical workflow trains a model on more capable hardware, converts it to a supported deployment format, then runs inference on the target device. Some systems support adaptation on-device, but that is a separate capability. LiteRT for Microcontrollers documents inference with a constrained operation set; it does not provide on-device training.
Measure four different budgets
A model file that fits in flash storage may still run out of RAM. Inference needs intermediate activations, input buffers, runtime state and memory for the rest of the application. Measure peak use on the real device. The model download size alone is not a deployment test.
Latency is the time until a useful result, including acquisition and preprocessing. Throughput counts work per unit of time; it does not guarantee that an individual command is answered promptly. Record typical and slow-case latency under realistic load. Repeated inference can also heat a device or drain its battery, changing sustained performance.
| Budget | Measure | Failure to investigate |
|---|---|---|
| Storage | Model, runtime and update space | New model cannot be installed safely. |
| Memory | Peak RAM while the full application runs | Inference fails after buffers are allocated. |
| Latency | Capture-to-result timing, including slow cases | A correct prediction arrives too late. |
| Energy | Consumption at the actual duty cycle | Continuous listening drains the battery. |
Convert first, then optimize
Check that the runtime supports the model’s operations, input shapes and target hardware. Establish a converted floating-point baseline before applying optimizations; otherwise a conversion problem and a quantization problem can be difficult to separate. Keep representative input/output pairs to compare each stage.
Quantization represents some values with lower precision, often reducing storage and enabling more efficient operations. It is not simply deleting parameters. Full integer conversion can require representative calibration inputs to estimate activation ranges. A calibration set containing only quiet recordings may poorly represent loud commands.
Accuracy and speed changes depend on the model, conversion mode and hardware. Do not promise a universal speedup. Recheck the false-activation and missed-command rates of the exact deployed artifact, including different speakers, microphones and background sounds.
See what lower precision changes
The following scalar illustration maps values to signed 8-bit integers with a fixed scale and zero point. It shows rounding and clipping, not a complete model converter. Run it in Node.js or a browser console. Real tools choose scales, may use per-channel values and have operator-specific rules.
The second input is outside the representable range. Its larger reconstruction error demonstrates why calibration and out-of-range inputs matter. A small numerical change can be harmless in one model and change a threshold decision in another.
const scale = 0.1;
function roundTrip(value) {
if (!Number.isFinite(value)) throw new TypeError('Finite input required');
const integer = Math.max(-128, Math.min(127, Math.round(value / scale)));
return integer * scale;
}
if (Math.abs(roundTrip(0.26) - 0.3) > 1e-10) throw new Error('Rounding');
if (Math.abs(roundTrip(20) - 12.7) > 1e-10) throw new Error('Clipping');
console.log(roundTrip(0.26).toFixed(1), roundTrip(20).toFixed(1));Expected output: 0.3 12.7. This demonstrates a numeric representation tradeoff; it does not predict accuracy or speed for a real model.
Design offline behavior deliberately
Local inference can avoid a network round trip, but other dependencies may still be remote: model download, licence checks, authentication, logging or a later action. Test after disconnecting the network, not only after disabling the model API. Make the interface distinguish “recognized locally” from “action completed remotely.”
For the keyword detector, a spoken command might be recognized while a connected service is unavailable. Keep the command pending only if that is safe, show its status and prevent duplicate execution when connectivity returns. Do not silently switch to a remote microphone upload as a fallback.
Release a reproducible device artifact
Record the model version, runtime, preprocessing configuration, input shape and evaluation results. Validate update integrity and compatibility before activating it. Keep a known working model so a failed rollout can be reversed. Model files are software supply-chain inputs, not automatically safe because they contain weights.
Edge inference can reduce raw-data transfer, but retained audio, crash reports and telemetry still need controls. Test where data actually flows and how users can stop collection. Compare the complete local design with a simpler rule or server design on measured requirements; “edge” is a placement choice, not a quality guarantee.
Try it yourself
Create a deployment report for a fictional keyword detector with separate storage, RAM, latency, energy and recognition-quality measurements. Add an offline test and a rollback criterion.
Show hint
Use clearly marked hypothetical numbers if you have no device. State how each value would be measured; do not infer runtime memory from file size or accuracy from quantization precision.
Check your understanding
4 questions to revisit the ideas. No score is saved, and you can retry. This is separate from Skill Quiz; it does not award XP or unlock guides.
Sources & further reading
Primary references for the concepts in this guide. Examples and exercises are illustrative; verify provider-specific behaviour against the version you use.