When to listen, when to stop, when it counts as done — the robot dialogue mechanism

Making a robot talk is easy. Making it hold a conversation is not.

Wire up speech recognition, an LLM and speech synthesis and you can have a robot answering you by the end of an afternoon. The hard part is what comes next: behaving like something you can actually negotiate with, in a real conversation that gets interrupted, goes off-topic, and happens while other things are going on.

This post is about how we do that in PhanthyMotus. The first half has no architecture in it — only the problems people actually hit in front of the robot. The second half covers the mechanisms, and which concrete failure forced each one.

What the user is waiting for

From the moment someone finishes their last syllable to the moment the robot starts making sound, this is the chain:

No single stage is slow. The VAD needs a stretch of silence to decide the sentence ended — our default is 400 ms — and transcription, inference and synthesis each cost a few hundred milliseconds to a few seconds. The problem is that they are in series and the user gets no feedback for the whole span. Not knowing whether the machine is listening, they repeat themselves — and the repeat becomes another input.

Latency is just the easiest complaint to voice. Three other things break the illusion much harder, and none of them is caused by the model being insufficiently clever:

One: you cannot get a word in. The robot is thirty seconds into an exhibit explanation and the visitor realises three sentences in that they asked the wrong thing. In a naive implementation they have to wait out the thirty seconds, because the system is busy and new input can only queue. Humans do not work that way: you start talking and the other person stops.

Two: a grunt becomes a command. Someone says something to the person next to them, or just makes an acknowledging noise. The microphone picks it up, ASR transcribes it, the model runs a full round of inference, and the robot answers a question nobody asked. In a room with several people this is not an edge case, it is the norm.

Three: the answer finishes before the action does. The robot says "certainly, I'll take you there", the audio finishes, and so the system considers the turn over — while it is in fact still walking. Finishing the sentence is not finishing the action, but as long as a turn ends when the model stops emitting, the two get conflated. The next utterance then gets processed while the action is still in flight, and the robot starts talking over itself.

What these three share: none of them is about what the model should say. They are about when to listen, when to stop, and when something counts as done — three moments that were never defined. That is what a dialogue mechanism is for.

Our approach: make the timing explicit

1. Not every event deserves to interrupt a conversation

A robot has many things producing events at once: microphone, battery, IMU, point cloud, scheduled tasks, chat messages, another robot on the LAN, and completion reports from its own actions. Feed all of that into one conversation context and the sensor stream washes out the model's attention — and every item costs an inference.

So the split happens before the main loop:

The criterion is priority. asr, channel, message, scheduler, acp and subagent are P>0 — they mean someone or something is addressing me — and go to the main agent. Sensor sources are P=0: they go to a background subagent on its own cadence, the main agent never sees them, and each source is throttled to 1 s. Everything lands in a ring buffer, so the model can query raw input when it wants it without passively drowning in it.

The point of this design: interrupting a conversation has a cost, so decide whether it is worth it first.

2. The user speaks while a turn is still running — three modes, "steer" by default

This is the most consequential choice in the whole mechanism. When a turn is in flight and the user starts talking, there are three possible responses:

steer is the default because what a user says mid-turn is usually a correction to the thing already underway, not a demand to throw it away. "Go to the second floor" followed by "take the stairs, not the lift" should join the current turn rather than restart it.

One small mechanism worth calling out: while busy, an ASR text already sitting in the queue is dropped. The repeat someone produced because they got no feedback should not become a second instruction.

3. Speech done ≠ action done: an explicit completion protocol

The third problem above comes down to this: a driver returning success usually means "command accepted", not "thing happened". When the TTS service call returns, audio has typically not started. When a navigation call returns, the robot has not taken a step.

We handle it with a completion protocol (ACP) around asynchronous actions: the action returns an action_id immediately so the model can keep reasoning, and before the next tool that acts, the harness automatically waits for every outstanding action to land. This layer is transparent to the model — it does not need to know asynchrony exists.

finish waits too, for a very specific reason:

If finish did not wait, a speak(...) followed by finish would end the turn before the audio played out, cutting the explanation short. But making finish wait resurrects the first problem: the default steer mode does not set cancel_event, and finish's wait spans the entire playback — so the queued utterance never gets consumed this turn and the user has to sit through the whole thing.

So that wait waits on two things: the action finishing, or a new message appearing in the steering queue. When the latter wins, playback is stopped, outstanding actions are cleared, finish is allowed through, the turn ends normally, and the message becomes the next turn's trigger. The user speaks, the robot stops — and not as a side effect that happens to work.

Stopping playback is not hardcoded to look for TTS, either. A driver declares x-hooks on its own tools (on_interrupt_speak, for instance) and the harness notifies whoever registered. Which tools count as "acting", and what should stop on an interrupt, is stated by the driver rather than guessed from names by the framework.

4. Layered context, so the stable part can be cached

Every turn rebuilds a prompt, and most of that prompt does not change. We split it four ways:

The assembly order is the part that matters: L1 + L2-static form the system message as a stable prefix, and L2-dynamic goes into the first user message instead. Dynamic content last means the prefix can hit the model's prefix cache. It is also why we did not end up expanding every peer robot's tools into the tool list — that grows the list with the fleet and busts the cache every time a peer comes or goes. Two fixed tools taking a peer id as an argument do the job instead.

Not solved yet

The short-backchannel filter does not actually fire. The design has a barge_in_threshold_ms (default 500 ms): while busy, an ASR event shorter than the threshold is treated as a backchannel and dropped. Today that check reads a field the event does not carry — perception emits audio_duration_ms with the whole JSON in the event's text, while the check reads payload.duration_ms, always gets 0, and the condition never holds. So problem two above is currently mitigated only by ASR deduplication and the VAD itself; the threshold is a knob that is not wired up. We will wire it, but we would rather not describe it as a capability we have.

No speaker separation. With several people in the room, "was that addressed to me" rests on the wake word and on content, with no voiceprint-level attribution.

Interruption is still coarse. We stop the current playback rather than stopping at a phrase boundary. When people interrupt each other, the one being interrupted usually finishes the phrase they are on.

In closing

There is no clever algorithm in any of this. It is mostly a matter of taking three moments that had been handled implicitly and making each of them explicit:

• When to listen — priority routing plus three interrupt modes, defaulting to folding mid-turn speech into the current turn;
• When to stop — making finish's wait wait on both "action played out" and "user spoke", with drivers declaring what an interrupt should stop;
• When it counts as done — a completion protocol separating "command accepted" from "thing happened", with an automatic wait before the next action.

Most of the difference in conversational feel does not come from which model you picked. It comes from whether these moments were defined with any care.

All three layers are open source: phanthymotus is Agent Core and the perception stack, phanthymotus-driver is the hardware drivers. The mechanisms above live in agent-core/src/collector.py, agent-core/src/event/llm.py and agent-core/src/prompt.py.