Voice Agents at the Edge - Reducing Latency in Real-Time AI Calls

Two hundred milliseconds is the difference between a voice agent that feels like a person and one that feels like a bad phone connection. Text chat tolerates a second of "thinking"; voice does not, because silence on a call reads as either a dropped connection or an awkward pause. Here's where that latency actually comes from and what shaves it down.

The Anatomy of a Voice Turn

A single conversational turn in a voice pipeline breaks down into stages, and each one adds latency you have to account for separately:

Caller stops speaking
   ↓ (network upload, ~20-60ms)
Speech-to-Text (STT) transcription
   ↓ (~150-400ms depending on model)
LLM time-to-first-token
   ↓ (~200-800ms depending on model + context size)
Text-to-Speech (TTS) synthesis
   ↓ (~100-300ms to first audio chunk)
Audio playback begins
   ↓ (network download, ~20-60ms)
Caller hears the response

Add it up naively and you're at 500ms-1.6s before the caller hears anything — and anything over ~500ms of dead air starts feeling wrong in a phone conversation. The entire discipline of voice AI latency engineering is finding where you can make these stages overlap instead of running them in strict sequence.

Where the Time Actually Goes

• Network round-trips — every hop between the caller's device, your media server, and each AI service adds fixed latency that no amount of model optimization fixes
• STT transcription — streaming STT models return partial transcripts continuously, but final-transcript confirmation (needed before you can safely start the LLM call) still costs real time
• LLM time-to-first-token (TTFT) — this is usually the single largest controllable chunk, and it grows with context length, so a voice agent with a bloated system prompt is paying a latency tax on every single turn
• TTS synthesis — non-streaming TTS waits for the full response text before generating any audio; streaming TTS can start speaking the first sentence while the LLM is still generating the rest

Edge and Regional Deployment

The fixed network latency floor is the one thing you can't optimize away with clever software — you can only get physically closer to the caller.

• Deploy your media/signaling server in the region closest to your caller base, not just your primary application region
• Use a provider with points-of-presence near major population centers for the SIP/WebRTC ingress layer specifically — this is often a different deployment decision than where your application logic runs
• Co-locate the STT and LLM inference endpoints in the same region as the media server; a call whose audio lands in us-east-1 but whose LLM call goes to eu-west-1 pays a cross-region penalty on every single turn, not just once

const regionForCaller = resolveNearestRegion(callerPhoneNumber);

const session = await voicePipeline.start({
  mediaRegion: regionForCaller,
  sttEndpoint: STT_ENDPOINTS[regionForCaller],
  llmEndpoint: LLM_ENDPOINTS[regionForCaller],
  ttsEndpoint: TTS_ENDPOINTS[regionForCaller],
});

Keeping all three AI services in the same region as the media edge is worth more than almost any model-level optimization, because it removes latency from every hop simultaneously.

Streaming Partial Responses

The single biggest lever after regional placement is refusing to treat STT → LLM → TTS as three sequential black boxes. Instead, stream at every boundary:

async function handleTurn(audioStream: AsyncIterable<AudioChunk>) {
  const transcriptStream = stt.streamTranscribe(audioStream);

  let partialText = '';
  for await (const chunk of transcriptStream) {
    partialText += chunk.text;
    if (chunk.isFinal) break;
  }

  const llmStream = llm.streamCompletion({
    model: 'claude-opus-4.6',
    messages: [...history, { role: 'user', content: partialText }],
  });

  const ttsQueue = new SentenceBoundaryQueue();

  for await (const token of llmStream) {
    ttsQueue.push(token);
    if (ttsQueue.hasCompleteSentence()) {
      const sentence = ttsQueue.flush();
      tts.streamSynthesize(sentence).pipeTo(audioOutput);
    }
  }
}

The key trick is the SentenceBoundaryQueue — instead of waiting for the full LLM response before calling TTS, you synthesize and start playing audio as soon as the first complete sentence is available. This alone typically cuts perceived latency by 40-60%, because the caller starts hearing a response while the LLM is still generating the rest of it.

Speculative and Early TTS Synthesis

More aggressive pipelines go a step further with speculative synthesis: start generating TTS audio for the most likely opening of the response before the LLM has even finished deciding on it, based on common response patterns for that intent, and discard it if the actual response diverges.

async def speculative_greeting(intent: str):
    likely_opener = COMMON_OPENERS.get(intent)
    if likely_opener:
        # Start synthesizing a probable opening phrase immediately,
        # in parallel with the real LLM call
        speculative_audio = asyncio.create_task(tts.synthesize(likely_opener))
        real_response = asyncio.create_task(llm.generate(intent))

        response_text = await real_response
        if response_text.startswith(likely_opener):
            return await speculative_audio  # cache hit, saved a full TTS round trip
        speculative_audio.cancel()

    return await tts.synthesize(await real_response)

This is a diminishing-returns optimization — it only pays off for a handful of highly predictable openers ("Sure, let me check that for you") — but for a high-volume voice agent, shaving even 100ms off the most common path adds up across millions of calls.

Measuring End-to-End Turn Latency

You cannot optimize what you don't measure at the right granularity. Track latency per stage, not just end-to-end, or you'll optimize the wrong thing:

interface TurnLatencyMetrics {
  audioEndToSttFinal: number;
  sttFinalToLlmFirstToken: number;
  llmFirstTokenToTtsFirstAudio: number;
  ttsFirstAudioToPlayback: number;
  totalPerceivedLatency: number;
}

function recordTurnLatency(metrics: TurnLatencyMetrics) {
  telemetry.histogram('voice.turn.stt_latency', metrics.audioEndToSttFinal);
  telemetry.histogram('voice.turn.llm_ttft', metrics.sttFinalToLlmFirstToken);
  telemetry.histogram('voice.turn.tts_first_audio', metrics.llmFirstTokenToTtsFirstAudio);
  telemetry.histogram('voice.turn.total_perceived', metrics.totalPerceivedLatency);
}

Track p50 and p95 separately for each stage — a system with a great median but a fat p95 tail will still feel broken to a meaningful fraction of callers, and the tail is usually where the real infrastructure problems (cold starts, regional mismatches, retry storms) hide.

Wrapping Up

Voice latency isn't one number to optimize, it's a pipeline of sequential stages that each have their own budget, and the wins come from overlapping those stages rather than making any single one faster in isolation. Get the region placement right, stream aggressively at every sentence boundary, and measure per-stage p95 — do those three things and a voice agent stops feeling like it's on a satellite delay and starts feeling like a conversation.