"Find the red cup" — open-vocabulary object detection on-device

Define Detection Targets With Language

Traditional object detection requires pre-trained fixed classes. Now you can dynamically tell the robot what to find.

How It Works

• Tell the robot: "Find the red cup"
• The Agent calls set_classes(["red cup"])
• YOLO + CLIP model detects it in the video feed in real time
• Returns the position and confidence when found

Technical Highlights

• **Open vocabulary** — Any English description can be detected, not limited to fixed classes
• **On-device inference** — Runs on Jetson CUDA GPU, no network required
• **Dynamic switching** — Change detection targets at any time without retraining
• **Pre-downloaded models** — Downloaded from COS on first boot, fully offline afterward

Use Cases

• Search and rescue: find a person wearing red clothing
• Warehousing: locate specific products
• Inspection: spot anomalous objects