Action models can now drive the robot directly
Drop an action model onto the canvas and the dozens of commands it emits every second now actually reach the robot's arms and grippers, with camera frames and joint state flowing back. Working end to end on Unitree G1 and Tianyi. When a model and a robot do not match, startup is refused with a plain statement of what differs.
A closed loop
An action model is not a chat model: it emits dozens of full action vectors per second and does not wait for an acknowledgement on each one. Until now the model card on the canvas could only idle — it connected, but nothing it produced reached the robot.
The loop is now closed:
camera frames + joint state → model → actions → arms / grippers → joint state back to the model
State is reported by the robot's own control card, so what the model sees and what it drives are the same numbers, in the same order, in the same units.
Mismatches are refused up front
Different models emit fundamentally different things — joint angles, joint velocities, or the position and orientation of the arm's end effector. They can have exactly the same number of dimensions.
That is the hardest kind of error to spot from symptoms: the dimensions line up, the rate lines up, startup succeeds, and then the arm moves on the wrong interpretation — it moves as if it made sense, and no layer reports anything.
So before startup the two sides are compared segment by segment: what the model says it emits versus what the robot says it accepts. A mismatch is refused, naming both sides and where each is configured. An undeclared space is also refused — the cost of guessing wrong is not an error message, it is an arm moving to the wrong place.
A missing input means no command that tick
Models declare how many cameras they need and whether they need joint state. That declaration is now checked at startup: pick a model and forget to wire up a camera, and you are told what is missing — instead of the card reporting "running" while never emitting a single command.
If an observation is missing at run time, no command is sent for that tick. Zeros are not substituted. Zero is a plausible reading in both units (arm straight, gripper open), and filling it in hands the model a world it cannot question. With no command, the robot holds the arm where it is — which is the correct state for "nothing is driving me right now".
Four actions the model can use
Execute, pause, interrupt, resume.
Pause and interrupt are not the same thing, and the difference is whether the already-planned chunk survives. The model emits an action chunk — a stretch of future plan. "Hold on" and "no, stop" are different instructions, and carrying out the second by replaying a plan made before the objection is exactly what the objection was rejecting. Switching tasks goes through interrupt for the same reason.
Robots this works on today
| Robot | Status |
|---|---|
| Unitree G1 | Streaming control of both arms plus both grippers, over the vendor's upper-body interface — no debug mode needed. Control is handed back with a weight ramp, so the arms never go limp and drop |
| Tianyi | Streaming control path now works end to end (the bridge previously ran outbound only, so commands never arrived) |
| RealMan RM75 | Joint-angle control |
Local SmolVLA inference is fixed too — it had never actually worked.