Sees faster and sharper — and can say how far away things are
Object recognition runs TensorRT directly on YOLOE-26s: 31.9 → 19.8 ms per frame on an Orin 5, accuracy from LVIS 24.4 to 30.8. A new visual_depth card estimates depth from an ordinary camera — in metres, not a relative scale — and a tape measure against a flat wall calibrates it to your own camera. Vision cards also answer about a single photo or URL with no camera attached.
Two things changed in the robot's vision: recognition got faster and more accurate, and it can now tell you roughly how many metres away something is — metres, not a vague "closer than that".
Faster and more accurate
Object recognition now drives a TensorRT engine directly, and the model moved from YOLOv8-World to YOLOE-26s.
Measured on an Orin 5 over the same frames (640 input, single frame, 144 classes): 31.9 ms per frame down to 19.8 ms, with accuracy up from LVIS 24.4 to 30.8. The new monocular depth model runs a frame in 12.5 ms.
One caveat: the list of recognizable classes is frozen when the engine is exported and cannot be changed at runtime. The card will tell you what it knows — call
list_recognizable_objects.
It can state distances now
The new visual_depth card estimates depth from an ordinary camera, filling the gap on robots with no depth sensor.
Depth readings are in metres. They used to be labelled "relative" with a warning that said "this is not metres" — which was wrong, and wrong in the more dangerous direction: tell an agent a reading is meaningless and it will never use it.
Making it yours: a tape measure and a flat wall
The model ships with a generic calibration, not a calibration of your camera. Three steps to fix that:
- Drive the robot squarely in front of a flat wall, lens facing it
- Measure the real lens-to-wall distance and pass it as
calibrate'sdistance_m - Back up and repeat at roughly 1 m, 2 m and 3 m
Got a measurement wrong? reset_calibration clears it and starts over — the fit is recomputed from all samples each time, so nothing drifts cumulatively. If the view is not actually flat (a corridor or a corner rather than a wall), or one sample clearly disagrees with the rest, the reply says so instead of quietly skewing the result.
Also
vop, ocr and visual_depth can now answer about a single photo or a URL, and the cards start without a camera attached. Asking "what's in this picture" no longer requires wiring up a live video path first.