Edge Inference: Running Vision Models on Robot Hardware

Running object detection on a mobile robot means solving a power-latency-cost triangle that no chip solves perfectly. You need inference fast enough for real-time decisions, low enough power to share a battery with motors and sensors, and a software stack that doesn't force you to rewrite your model. The three platforms that dominate mobile robotics—NVIDIA Jetson, Google Coral, and Rockchip RK3588—each make different trade-offs.

Why Edge Inference Matters for Navigation

Cloud inference adds 50–200 ms of round-trip latency, which is unusable for obstacle avoidance at walking speed. A robot moving at 1 m/s travels 20 cm while waiting for a cloud response at 200 ms—enough to miss a step or collide with a moving object. Edge inference cuts that to 10–50 ms, depending on the model and chip, and eliminates dependency on network uptime.

The challenge is fitting a useful vision model into a 5–25 W thermal budget on a platform that also powers motors, lidar, and a compute board. Most mobile robots reserve 10–15 W for the inference accelerator, which rules out desktop GPUs and forces you into purpose-built edge chips.

NVIDIA Jetson: Flexibility at the Cost of Power

Jetson modules (Nano, Xavier NX, Orin Nano, Orin NX) offer the widest model compatibility because they run CUDA and TensorRT, the same stack as datacenter GPUs. You can deploy YOLOv8, EfficientDet, or custom PyTorch models with minimal conversion overhead. The Orin Nano delivers 40 TOPS (tera-operations per second) at 15 W, enough to run YOLOv8-medium at 640×640 resolution at 30 fps with 12–18 ms inference time.

The downside is power. Even the most efficient Jetson module draws 7–10 W at idle and 12–20 W under load, which can halve battery life on small mobile platforms. Jetson also requires active cooling—usually a small fan—which adds noise and another failure point. For indoor service robots or research platforms where flexibility matters more than runtime, Jetson is the default choice.

Google Coral: Maximum Efficiency, Narrow Model Support

The Coral Edge TPU (USB accelerator or M.2 module) runs quantized TensorFlow Lite models at 4 TOPS and draws only 2–3 W under load. A MobileNet SSD v2 model at 300×300 resolution runs at 60 fps with 7–10 ms latency per frame, making it the lowest-power option for simple detection tasks. The entire module is passively cooled and costs under $60.

The constraint is model support. Coral only accelerates models compiled for its Edge TPU, and not all layer types are supported. YOLOv5 works with some manual surgery; YOLOv8 requires re-training or switching to a supported architecture. If your application fits a pre-trained MobileNet or EfficientNet-Lite model, Coral offers the best performance-per-watt. If you need custom architectures or frequent model updates, the conversion friction becomes a liability.

Rockchip RK3588: The Emerging Middle Ground

The RK3588 is a system-on-chip with a 6 TOPS NPU (neural processing unit) and eight CPU cores, commonly found on boards like the Orange Pi 5 or Radxa Rock 5B. It runs RKNN models (Rockchip's format) and supports ONNX conversion, giving it broader compatibility than Coral but less than Jetson. Power draw sits between the two: 5–10 W under full load, with passive cooling sufficient for most use cases.

Real-world performance for YOLOv5s at 640×640 is around 25–35 ms per frame, which is adequate for navigation but slower than Jetson. The software stack is less mature—expect to spend time debugging ONNX-to-RKNN conversion and tuning quantization settings. For cost-sensitive projects or applications where Jetson's power draw is prohibitive and Coral's model constraints are too tight, RK3588 offers a practical compromise.

Real-World Latency Benchmarks

Latency depends on resolution, model size, and quantization. The table below shows typical end-to-end inference times for common detection models across the three platforms, including preprocessing and postprocessing on the host CPU.

For navigation, aim for total perception-to-decision latency under 50 ms. If inference takes 30 ms, you have 20 ms left for tracking, path planning, and motor commands. Faster models (MobileNet) sacrifice accuracy for speed; larger models (YOLOv8-medium) improve detection of small or distant objects but require more compute.

Choosing the Right Platform

Start with your model and power budget. If you already have a PyTorch or TensorFlow model and need to iterate quickly, Jetson removes friction. If your robot runs on a small battery and you can commit to a supported architecture, Coral maximizes runtime. If cost or power sits between those extremes and you can tolerate toolchain rough edges, RK3588 is worth testing.

Thermal management is non-negotiable. Even passive heatsinks require airflow; enclosed robots need ventilation or a small fan. Monitor junction temperature under sustained load—if the chip throttles, your inference time will spike unpredictably, which breaks navigation assumptions.

Software Stack and Model Optimization

All three platforms benefit from quantization—converting 32-bit floating-point weights to 8-bit integers. Jetson uses TensorRT for automatic quantization with minimal accuracy loss. Coral requires full integer quantization during TFLite conversion, which can drop mAP (mean average precision) by 2–5 points if not carefully calibrated. RK3588 uses RKNN-Toolkit, which supports mixed precision but demands manual tuning.

Test quantized models against a validation set that matches your deployment environment. Indoor robots need strong performance on low-contrast objects and partial occlusions; outdoor robots prioritize distant detection and tolerance to lighting changes. A model that benchmarks well on COCO may underperform in your specific use case.

For teams building smaller robots with integrated language models, edge inference can be combined with NLP accelerators, though power budgets become tight. Understanding the broader hardware ecosystem helps contextualize where vision fits into the robot's compute allocation.

Future-Proofing Your Hardware Choice

Edge AI chip performance doubles roughly every 18 months, but software ecosystems take years to mature. Jetson has the longest support cycle and the largest community; Coral is stable but no longer actively developed; RK3588 is newer and still evolving. If your product has a multi-year lifecycle, plan for a second-generation hardware refresh and validate that your model pipeline can migrate between platforms.

Avoid locking yourself to a single vendor's proprietary format. Train in PyTorch or TensorFlow, export to ONNX as an intermediate representation, then convert to platform-specific formats. This lets you benchmark multiple chips without retraining and gives you an exit path if a platform is discontinued.

For more context on how different regions approach robotics development and funding, see our analysis of US and international robotics policy.