Faster to respond, faster to recover
LLM calls are streamed, so a stuck connection and a long answer are finally distinguishable. Liveness is judged on time-to-first-token (10/20/60 s, rising per retry) and the gap between tokens, cutting the worst case from six minutes to 90 seconds — while the total budget actually loosens to 600 s so long answers are no longer killed by mistake.
Occasionally a request hangs. The robot used to sit through a full two minutes before retrying — worst case three times over, six minutes burned. Worst case is now 90 seconds.
The problem was not noticing late — there was nothing to notice
Nothing came back until the entire response did, which made "the request is stuck" and "the answer is long and still generating" indistinguishable until the timeout fired.
An actual incident: a large request went out at 12:17:38. The same model was called three times alongside it and answered in seconds each time. The large request sat there until it timed out at 12:19:38 — and a retry of the identical payload succeeded in 7 seconds. The model was not slow and the service was not down; one connection was stuck, and there was no way to find that out sooner.
With streaming
The answer now arrives token by token, so the test changes:
| What is measured | Threshold |
|---|---|
| Time to the first token | 10 s / 20 s / 60 s, rising per retry |
| Gap between tokens | 10 s |
| Total response time | 600 s |
The first-token budget escalates rather than sitting at a fixed value: probe at 10 s (a stuck connection recovered in 7 s when measured), and if it really is just slow, the second and third attempts let it through.
The total budget actually got looser. The old 120 s was a cap on how long an answer could take to generate, so long answers were killed by mistake. Liveness is now judged by the gap between tokens, and length is no longer grounds for suspicion.