Give your robot a test run with a score at the end
Run a fixed test case as a full task and get a score at the end: did it go to the right places, how long did each round of thinking take, was the user left waiting, were there any unsafe moves. It used to run only in a virtual world; it now runs on real robots too, judged by the same rules. Afterwards you can lay out what the agent asked for and what the world actually did on one shared timeline.
A test case is a whole solution
A test case is not a button on some card — it is a complete solution: how the canvas is wired, what the task is, and what counts as passing. It ships through the same channel as skills and solutions, so you can pull one from the marketplace or write your own.
When a case cannot run, the reason is layered: which driver is missing → which map is missing → the case itself is malformed. Each layer implies a different next step, rather than a blanket "unavailable".
Seven principles, measured wherever measurement is possible
"Was the user experience good" is a vibe to a model. So every principle starts from a computable metric, and the model only judges on top of those:
| What is looked at | How it is measured |
|---|---|
| Physical-world ordering | Did it leave before finishing what it was saying, do reports contradict the facts, how many navigation legs |
| Execution efficiency | Total duration, time in inference versus tools, parallelism, idle share |
| Inference latency | Median / P95 / slowest round |
| Cache hits | The hit rate is itself the score, not a pass threshold |
| Answer quality | Judged by the model — this one genuinely has no computable metric |
| User experience | Longest and mean silent-thinking time, first response, response after an interruption |
| Safety | Collisions and intrusions, combined into an incident count |
The model judges only three things: answer quality, whether a deviation from the reference flow was reasonable, and the plain-language requirements written into the case. It has every metric in hand while doing so, so it is a grounded judgement rather than a gut score.
Dimensions that cannot be computed are not scored. One safety dimension on real robots was always missing its input and therefore silently scored a perfect 100 — a perfect score for something never checked. It is now excluded rather than faked.
What it wanted, and what happened
The run detail lays the scene out: one shared timeline, the agent's requests on the left, what the world actually did on the right, the same second always on the same row.
Every stretch of silence is drawn. A stall, a spinning model and a timeout all look identical in a log, and that blank stretch is usually the answer.
The same judging on real hardware
Judging used to live inside the virtual world — the referee living inside the system under test, where a broken path could score itself green. The referee is now independent: a real robot only has to emit the same shape of fact stream and the same assertions apply.
Two differences are stated explicitly rather than hidden behind the score: the physical world has no spawn point to return to, so a second repetition starts where the first one ended; and navigating by coordinates has no station names, so the "station order" check fails — which is the correct verdict.
Anything that moves needs your nod
A test case's instructions look exactly like your own to the robot. So before a run you get a list of everything that will physically move, for a person on the spot to confirm — and the server recomputes and re-checks that list at launch, because the canvas can change between the dialog and the run. The list appears while you are choosing a case, not as a single gate after you press run.