Serverless LLM Inference - AWS Lambda, API Gateway, and Bedrock Streaming

Running a persistent GPU server for LLM inference makes sense when you own the model weights; it makes a lot less sense when the actual work is just proxying to Bedrock and streaming tokens back to a browser. For that shape of workload, a Lambda that scales to zero and bills per invocation is usually the better tradeoff — as long as you get response streaming right, because a buffered Lambda response turns a nicely streaming Bedrock call into a client that waits in silence for the whole answer.

Lambda Response Streaming, Not Buffering

By default, Lambda buffers the entire response before returning it. awslambda.streamifyResponse changes that, letting you write chunks to the client as they arrive from Bedrock.

import { BedrockRuntimeClient, ConverseStreamCommand } from '@aws-sdk/client-bedrock-runtime';
import { Readable } from 'node:stream';

const bedrock = new BedrockRuntimeClient({ region: 'us-east-1' });

export const handler = awslambda.streamifyResponse(
  async (event, responseStream, _context) => {
    const { prompt } = JSON.parse(event.body);

    const httpResponseMetadata = {
      statusCode: 200,
      headers: { 'Content-Type': 'text/event-stream' },
    };
    responseStream = awslambda.HttpResponseStream.from(responseStream, httpResponseMetadata);

    const command = new ConverseStreamCommand({
      modelId: 'anthropic.claude-4-5-haiku-v1:0',
      messages: [{ role: 'user', content: [{ text: prompt }] }],
    });

    const { stream } = await bedrock.send(command);
    for await (const event of stream ?? []) {
      const delta = event.contentBlockDelta?.delta?.text;
      if (delta) {
        responseStream.write(`data: ${JSON.stringify({ token: delta })}\n\n`);
      }
    }
    responseStream.end();
  },
);

The client sees tokens arrive as Bedrock produces them instead of one large payload after the full generation completes — the same perceived-latency win you'd get from a persistent server, without keeping a process warm and idle between requests.

Exposing It: Function URLs vs. API Gateway

Streaming responses need a front door that doesn't buffer them either. Lambda Function URLs support RESPONSE_STREAM invoke mode directly:

aws lambda create-function-url-config \
  --function-name bedrock-chat-stream \
  --auth-type AWS_IAM \
  --invoke-mode RESPONSE_STREAM

API Gateway (HTTP API) works too, but as of this writing it sits in front of Lambda without native streaming pass-through the way a Function URL does — most teams put HTTP API in front for auth, throttling, and custom domains, and route only the streaming endpoint through a Function URL, fronted by CloudFront for TLS and caching of the static parts of the app.

Client
  │
  ├── /api/*        → API Gateway (HTTP API) → standard Lambda (buffered, fine for CRUD)
  └── /api/chat      → CloudFront → Lambda Function URL (RESPONSE_STREAM) → Bedrock

Cold Starts on an AI Endpoint

Cold starts matter more here than on a typical CRUD Lambda, because the AWS SDK's Bedrock client and any embedding model dependencies add real init time on top of the runtime's own startup.

// Keep the Bedrock client outside the handler — reused across warm invocations
const bedrock = new BedrockRuntimeClient({ region: 'us-east-1' });

export const handler = awslambda.streamifyResponse(async (event, responseStream) => {
  // handler body only does per-request work
});

Provisioned Concurrency removes cold starts entirely for a predictable baseline of traffic, at the cost of paying for idle capacity — which mostly defeats the "scale to zero" appeal that made serverless attractive in the first place. In practice, for a chat endpoint with bursty but not constant traffic, a few provisioned instances sized to typical p50 concurrency plus on-demand scaling for spikes lands closer to the sweet spot than either extreme.

Cost: Serverless vs. a Persistent Server

For a workload that proxies to a managed model API rather than hosting weights, the comparison isn't GPU-hours vs. Lambda-seconds — it's idle-server cost vs. per-invocation cost.

The rough breakeven in practice: once sustained concurrent requests are high enough that a fixed-size fleet is busy most of the day, the per-invocation Lambda pricing starts costing more than just running the servers. Below that line — which covers a surprising number of internal tools and early-stage products — serverless wins outright, and the code stays simpler because there's no autoscaling group to tune.

Wrapping Up

The mechanics here aren't new — Lambda has supported response streaming for a while, and API Gateway has always been able to front a Lambda. What makes this pattern worth reaching for specifically now is that Bedrock's own streaming API lines up cleanly with Lambda's streaming response model, so you get token-by-token delivery to the client without running a single always-on process. Reach for a persistent server when traffic is constant and sustained; reach for this stack when traffic is spiky and the alternative is paying for GPU-adjacent compute that sits idle most of the day.