Voice agent architecture: latency, interruptions and state

A voice agent is a pipeline of separate systems: telephony, speech-to-text, a language model, tools the model can call, and text-to-speech. A demo works when every stage is fast and the caller waits politely. A real call has background noise, a caller who talks over the agent, a model provider that slows down, and a connection that drops. This note covers the four parts that decide whether the agent holds up: the latency budget, streaming, interruptions and call state, and fallbacks.

Set a latency budget for each stage

Callers notice a pause after they stop speaking much sooner than they notice a slow web page. Pick a target for the time from the end of the caller's speech to the first audio of the reply, then split it across the stages, so each stage has a number it can be measured against.

For example, if the target is about one second, a budget might look like this:

These numbers are an illustration, not a benchmark. The point is to measure every stage separately, at the median and at the 95th percentile, from real calls. An average hides the slow calls, and the slow calls are the ones callers remember.

Tool calls usually break the budget. If the agent has to look up an order before it can answer, say something short first ("Let me check that order") while the lookup runs.

Stream, but speak in whole phrases

Waiting for the full model response before speaking adds the whole generation time to every turn. Streaming fixes that, but sending each token to text-to-speech as it arrives makes things worse: single tokens are often fragments of words, the voice loses its intonation, and many providers add overhead per request.

Buffer the stream and send whole phrases. Flush at the end of a sentence, or at a clause boundary once enough text has built up.

// End of a sentence or clause, followed by whitespace. Requiring the whitespace keeps
// "3.5" in one piece, because the decimal point is followed by a digit, not a space.
const BOUNDARY = /[.!?;:,]["')\]]?\s+/g;
const MIN_PHRASE_CHARS = 24; // shorter fragments sound clipped, so keep buffering

async function speakAsItStreams(tokens: AsyncIterable<string>, tts: { enqueue(text: string): void }) {
  let buffer = '';
  for await (const token of tokens) {
    buffer += token;
    let cut = 0;
    for (const m of buffer.matchAll(BOUNDARY)) {
      const end = (m.index ?? 0) + m[0].length;
      if (end >= MIN_PHRASE_CHARS) cut = end;
    }
    if (cut > 0) {
      tts.enqueue(buffer.slice(0, cut).trim());
      buffer = buffer.slice(cut);
    }
  }
  if (buffer.trim()) tts.enqueue(buffer.trim()); // whatever is left when the model stops
}

Two details matter in practice. Abbreviations such as "Dr." and "e.g." end in a period followed by a space, so keep a short list of them and skip those boundaries. And some text-to-speech providers accept a text stream and do this segmentation themselves; if yours does, use it and measure the first-audio time it gives you.

Handle interruptions

Callers talk over the agent. When the speech-to-text stage detects the caller speaking during playback:

1. Stop playback within a few hundred milliseconds.
2. Cancel the queued text-to-speech requests and the model stream that feeds them.
3. Record what was spoken before the cut, not what was generated.

The third step is the one most often missed. If the conversation history contains the full generated reply, the model believes the caller heard things they did not, and the next turn goes wrong. Track playback position, and write only the spoken part of the reply into the history.

Keep call state outside the process

A call can outlive the process that started it: a deploy restarts the worker, or a reconnect lands on a different one. Keep the state of each call in a shared store with an expiry, and keep enough in it to resume the call.

interface CallState {
  callId: string;
  turns: { role: 'caller' | 'agent'; text: string; spokenChars?: number }[];
  toolResults: { name: string; result: unknown; at: string }[];
  pendingAction?: { kind: string; payload: unknown; confirmed: boolean };
}

// Expire state an hour after the last update. Long calls refresh it on every turn.
await redis.set(`call:${state.callId}`, JSON.stringify(state), 'EX', 3600);

spokenChars records how much of each agent turn was heard before an interruption. pendingAction holds anything the agent intends to change, such as a booking, until the caller has confirmed it.

Confirm before anything is written

Reading is cheap to get wrong; writing is not. Before the agent books an appointment, changes an order or updates a customer record, it reads the change back and waits for a clear yes. Changes that are expensive to undo go to a person instead. Every action, confirmed or not, goes into the call's log with the turn that triggered it.

Fail over without going silent

Model and voice providers slow down and fail. Give every stage a timeout that fits its budget, try a second provider when the first times out, and keep a scripted reply and a transfer to a person as the last step.

const providers = [primaryModel, secondaryModel];

async function reply(prompt: Prompt, intent: Intent): Promise<Reply> {
  for (const provider of providers) {
    try {
      return await provider.generate(prompt, { signal: AbortSignal.timeout(1500) });
    } catch (err) {
      log.warn('model provider failed', { provider: provider.name, err });
    }
  }
  // Nothing answered in time: say something useful and hand the call to a person.
  return { text: scriptedReply(intent), transferToHuman: true };
}

A second model behaves differently from the first, so it needs its own run through the evaluation set before it can take live calls.

Test on recorded calls

Build an evaluation set from real calls, recorded with the caller's consent, with the correct outcome written down for each. Rerun it before every change to a prompt, a tool, a model or a voice. Track the stage latencies, how often callers interrupt, how often the agent transfers to a person, and how often a confirmed action turns out to be wrong. This is the same evaluation-first method we use on our Production AI agent platform, applied to a pipeline where every stage has a clock on it.