Voice AI Chatbots - Combining ElevenLabs, Deepgram, and LLMs for Natural Conversations

The gap between a voice bot that sounds impressive in a controlled demo and one that survives a real phone call comes down almost entirely to latency and interruption handling. Users don't wait politely for a full sentence to finish transcribing, then a full LLM response to generate, then a full audio clip to render — they talk over silence, change their mind mid-sentence, and hang up if the pause after they stop speaking feels even slightly wrong. Getting a voice pipeline to feel natural means treating every stage as a stream, not a request-response call.

Streaming STT with Deepgram

Deepgram's WebSocket API returns partial transcripts continuously, not just a final result at the end of an utterance.

import { createClient, LiveTranscriptionEvents } from '@deepgram/sdk';

const deepgram = createClient(process.env.DEEPGRAM_API_KEY!);
const connection = deepgram.listen.live({
  model: 'nova-3',
  smart_format: true,
  interim_results: true,
  endpointing: 300, // ms of silence before marking utterance final
  vad_events: true,
});

connection.on(LiveTranscriptionEvents.Transcript, (data) => {
  const transcript = data.channel.alternatives[0].transcript;
  if (!transcript) return;

  if (data.is_final) {
    handleFinalTranscript(transcript);
  } else {
    handlePartialTranscript(transcript); // fires many times per second
  }
});

connection.on(LiveTranscriptionEvents.SpeechStarted, () => {
  interruptCurrentPlayback(); // barge-in signal
});

endpointing controls how long the engine waits for silence before deciding the user is done — set it too low and it cuts people off mid-thought, too high and every response feels sluggish. 250-400ms is a reasonable starting range, tuned per use case.

Early Intent Detection from Partial Transcripts

Waiting for the full final transcript before calling the LLM wastes the most valuable seconds of the interaction. Feeding partials to a lightweight intent classifier lets you start preparing a response before the user finishes talking.

async def on_partial_transcript(partial: str, session: VoiceSession):
    intent_guess = await fast_intent_classifier.predict(partial)

    if intent_guess.confidence > 0.85 and intent_guess.label == "check_order_status":
        # Pre-fetch likely-needed data while the user keeps talking
        session.prefetch_task = asyncio.create_task(
            order_service.get_recent_orders(session.user_id)
        )

By the time the final transcript arrives and the full LLM call fires, the order lookup is often already resolved, shaving several hundred milliseconds off perceived response time — the part users actually notice.

Streaming TTS with ElevenLabs for Low Time-to-First-Audio

The LLM response itself should stream token-by-token into ElevenLabs rather than waiting for the full text, so the first audio chunk plays while the model is still generating the rest of the sentence.

import { ElevenLabsClient } from 'elevenlabs';

const elevenlabs = new ElevenLabsClient({ apiKey: process.env.ELEVENLABS_API_KEY! });

async function speakStreamingResponse(llmTokenStream: AsyncIterable<string>) {
  const audioStream = await elevenlabs.textToSpeech.convertAsStream(VOICE_ID, {
    model_id: 'eleven_flash_v2_5', // optimized for low latency over max fidelity
    text: textFromChunkedStream(llmTokenStream),
    optimize_streaming_latency: 4,
    voice_settings: { stability: 0.4, similarity_boost: 0.75 },
  });

  for await (const chunk of audioStream) {
    playbackQueue.push(chunk); // starts playing before generation finishes
  }
}

Chunking text into sentence-sized pieces before sending to TTS (rather than one giant final blob) is what actually gets time-to-first-audio down into the 200-400ms range instead of waiting out the full LLM generation first.

Natural Conversation Touches

A technically correct pipeline can still sound robotic. A few details close most of that gap:

• Filler words during processing — inject a short "let me check that" utterance while a slow tool call runs, instead of dead air
• Interruption handling (barge-in) — the moment SpeechStarted fires from Deepgram, immediately stop playback and discard the in-flight TTS stream rather than letting it finish
• Backchannel acknowledgment — brief "mm-hm" style audio cues during long user utterances, generated from a small pre-rendered clip library rather than a live model call
• Prosody-aware chunking — break TTS requests at clause boundaries, not arbitrary character counts, so intonation doesn't sound chopped

Wiring It All Together

class VoiceSession {
  async handleAudioChunk(chunk: Buffer) {
    this.deepgramConnection.send(chunk);
  }

  private async onFinalTranscript(text: string) {
    this.currentTtsStream?.abort();
    const llmStream = await llmClient.streamChat({
      messages: [...this.history, { role: 'user', content: text }],
    });

    let sentenceBuffer = '';
    for await (const token of llmStream) {
      sentenceBuffer += token;
      if (/[.!?]\s*$/.test(sentenceBuffer)) {
        this.ttsQueue.enqueue(sentenceBuffer);
        sentenceBuffer = '';
      }
    }
  }

  private onBargeIn() {
    this.ttsQueue.clear();
    this.audioOutput.stop();
  }
}

The three vendors never need to know about each other directly — Deepgram emits transcript events, an orchestrator turns those into LLM calls and TTS requests, and ElevenLabs emits audio chunks. Keeping that boundary clean is what lets you swap any one piece (a different STT provider, a different model) without rewriting the interruption and streaming logic that took the longest to get right in the first place.

Wrapping Up

Every one of these techniques exists to attack the same number: the silence between a user finishing a sentence and hearing a relevant reply. Streaming STT, partial-transcript intent detection, sentence-level TTS chunking, and aggressive barge-in handling each shave off a piece of that gap, and stacked together they're the difference between a voice bot that feels like a phone tree with better words and one that feels like it's actually listening.