A text chatbot can get away with a two-second response delay. A voice bot cannot - anything past roughly 800ms of silence and the human on the other end starts talking again, assumes the call dropped, or just hangs up. Building a voice AI agent is really an exercise in latency budgeting across three separate systems (speech-to-text, the LLM, and text-to-speech) that were never designed to talk to each other in real time. Here's how we wire that up with LiveKit.
The STT → LLM → TTS Pipeline
Every real-time voice agent is the same three stages glued together:
- STT (speech-to-text) - streaming transcription of the caller's audio, producing partial and final transcripts
- LLM - reasoning over the transcript plus conversation history to produce a response
- TTS (text-to-speech) - streaming synthesis of the response back into audio frames
The naive approach - wait for a complete utterance, wait for a complete LLM response, then synthesize a complete audio clip - adds up to 3-5 seconds of dead air per turn. Nobody tolerates that. Production voice bots pipeline all three stages so each one starts working on partial output from the one before it.
Caller audio ──▶ STT (streaming) ──▶ partial transcript ──▶ LLM (streaming)
│
partial tokens
│
Caller ◀── audio frames ◀── TTS (streaming) ◀── sentence-chunked text
Why Streaming Transcripts Matter for Latency
Waiting for STT to emit a "final" transcript before invoking the LLM throws away 200-400ms of usable time. Instead, we start speculative LLM generation on stable partial transcripts and only commit to a response once STT marks the utterance final and a short silence timeout passes (typically 300-500ms of trailing silence, tuned per use case).
stt.on('transcript', (event) => {
if (event.isFinal) {
silenceTimer = setTimeout(() => {
commitTurn(event.transcript);
}, 400);
} else {
// Cancel any pending speculative response if the caller keeps talking
clearTimeout(silenceTimer);
}
});
On the TTS side, we don't wait for the full LLM response either. We chunk the streamed token output on sentence boundaries and push each sentence to the TTS engine as soon as it's complete, so audio starts playing while the LLM is still generating the rest of the answer.
let buffer = '';
for await (const delta of llmStream) {
buffer += delta.text;
const sentenceMatch = buffer.match(/^(.+?[.!?])\s/);
if (sentenceMatch) {
const sentence = sentenceMatch[1];
buffer = buffer.slice(sentence.length).trimStart();
ttsQueue.push(sentence); // synthesize and stream immediately
}
}
if (buffer.trim()) ttsQueue.push(buffer.trim());
Handling Interruptions and Barge-In
Real conversations are not turn-based - people interrupt. If the caller starts speaking while the bot is still talking, you need to detect it (voice activity detection on the inbound track), immediately stop TTS playback, cancel the in-flight LLM generation, and re-open the mic for a new turn.
vad.on('speechStart', () => {
if (agentState === 'speaking') {
ttsPlayer.stop();
llmController.abort();
agentState = 'listening';
}
});
Getting barge-in wrong is the single biggest tell that a voice bot is "just a demo." Callers expect to be able to cut in with "no wait, I meant next Tuesday" and have the bot actually stop and listen.
LiveKit Room and Track Architecture
LiveKit gives you a WebRTC-based room abstraction: a caller (or a SIP trunk bridging a phone call) publishes an audio track into a room, and your agent runs as a headless participant that subscribes to that track and publishes its own synthesized audio track back.
from livekit.agents import JobContext, WorkerOptions, cli
from livekit.agents.voice import VoiceAgent
from livekit.plugins import deepgram, openai, elevenlabs, silero
async def entrypoint(ctx: JobContext):
await ctx.connect()
agent = VoiceAgent(
vad=silero.VAD.load(),
stt=deepgram.STT(model="nova-3", interim_results=True),
llm=openai.LLM(model="gpt-4.1-mini"),
tts=elevenlabs.TTS(voice="rachel", streaming_latency=2),
)
@agent.on("user_speech_committed")
def on_user_speech(transcript: str):
print(f"caller said: {transcript}")
await agent.start(ctx.room)
if __name__ == "__main__":
cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))
LiveKit's agent worker model spins up one job per room, so scaling horizontally is just running more worker processes behind the same signaling server - no custom autoscaling logic needed for the media path itself.
Telephony Integration
For phone-based voice bots, LiveKit's SIP bridge terminates the inbound call and publishes it into a room exactly like a WebRTC participant would, which means the same agent code handles web-based and phone-based calls without a fork in your pipeline.
livekit-cli sip create \
--trunk-id "trunk_inbound_support" \
--dispatch-rule '{"type":"individual","roomPrefix":"support-call-"}'
Latency Budget, End to End
A rough budget that keeps turn-taking feeling natural:
- STT partial transcript: under 150ms
- Silence/endpointing detection: 300-500ms
- LLM time-to-first-token: under 400ms
- TTS time-to-first-audio-byte: under 200ms
- Network/jitter buffer overhead: 50-100ms
Stay under roughly 1.2 seconds total and most callers perceive the bot as responsive rather than robotic.
Observability for Voice Agents
Text logs aren't enough - you need to correlate transcript events, LLM latency, and TTS latency against the actual audio timeline to debug why a specific call felt slow. We tag every pipeline stage with the same turnId and ship structured events to our tracing backend, then reconstruct a waterfall per call for postmortems.
Voice AI in 2026 isn't about picking the smartest model - GPT-4.1-mini or Claude Haiku both do fine for most conversational tasks. It's about disciplined pipelining, aggressive interruption handling, and never letting any single stage block the other two. Get the plumbing right and the "AI" part almost takes care of itself.