Streaming LLM answers: the proxy that held the words back

A model takes a few seconds to write a hundred-word answer. Showing those words as they come changes nothing about that duration, but everything about how it feels: the first word appears in under a second, and reading takes longer than writing anyway. It is the most cost-effective performance optimisation a chat interface can get, and it does not touch the model.

Except that between the model and the screen there is your server, sometimes a compression middleware, often a proxy, and each of them can decide to keep the words to itself. The symptom is always the same. Locally, everything appears progressively. In production, the answer arrives in one block after several seconds of empty screen.

The trip of a word

Most model APIs send their answer as Server-Sent Events: an HTTP connection that stays open, and text events sent as generation goes, each ending with a blank line. The format is tiny. A line starting with a colon is a comment the client ignores, which will come in handy.

event: delta
data: {"text": "Code E-07"}

event: delta
data: {"text": " means the speed sensor"}

: ping

event: done
data: {}

The simulation follows one answer from the model to the browser. Turn the intermediaries on one by one, then replay: the total duration barely moves, the moment the first word appears does.

The buffering proxy

By default, nginx reads the server’s response into its own buffers before passing it on: that is proxy_buffering on. For an HTML page it is a good idea, as it frees the application server sooner. For a stream it is the failure described above: the words wait in the proxy until the response ends.

The cleanest fix lives in the application: send the X-Accel-Buffering: no header on streaming responses. nginx understands it and turns buffering off for that response only, without touching the rest of the site. If you prefer configuration, you can also isolate the route:

location /api/chat {
    proxy_pass         http://app;
    proxy_http_version 1.1;
    proxy_buffering    off;   # or send X-Accel-Buffering: no from the app
    proxy_read_timeout 300s;  # the default closes after 60 s of silence
    gzip               off;
}

The compression that waits

A compression middleware needs material to compress. Express’s compression holds data until its buffer fills, unless the application calls res.flush() after each event. The simplest option is often to exclude text/event-stream from compression: a few kilobytes of text are not worth the added latency.

The silence that cuts

An agent does not talk continuously. When it calls a slow tool, a search or a database query, the stream can stay silent for a long time. And intermediaries close silent connections: 60 seconds by default for nginx’s proxy_read_timeout, 60 seconds as well for an AWS load balancer’s idle timeout. Turn on the tool call in the simulation: with no sign of life, the connection drops and the end of the answer never arrives.

The cure is a heartbeat: an SSE comment line, sent every fifteen seconds. The client ignores it, the proxies see traffic. It is also a chance to send a status event, “searching the documentation”, which tells the user what is going on during the wait instead of leaving them in front of a frozen cursor.

In the browser

EventSource, the native API, only does GET requests, with no body and no custom headers. To POST a conversation with a token, read the stream with fetch and a ReadableStream. You lose automatic reconnection along the way, so you handle it yourself.

const res = await fetch('/api/chat', {
  method: 'POST',
  headers: { 'content-type': 'application/json' },
  body: JSON.stringify({ messages }),
  signal: controller.signal // the Stop button aborts the whole chain
})
const reader = res.body!.pipeThrough(new TextDecoderStream()).getReader()
let buffer = ''
for (;;) {
  const { value, done } = await reader.read()
  if (done) break
  buffer += value
  // events end with a blank line; keep the unfinished tail for the next chunk
  const events = buffer.split('\n\n')
  buffer = events.pop() ?? ''
  for (const event of events) {
    const line = event.split('\n').find((l) => l.startsWith('data: '))
    if (line) onEvent(JSON.parse(line.slice(6)))
  }
}

The splitting deserves care: a chunk from the network can hold several events, or half of one. Accumulate, split on blank lines, and keep the incomplete tail for the next chunk.

What the interface has to get right

Once the words really arrive, the rest happens in the interface. These are small details, and they are what makes an answer feel fluid rather than like text fighting its way onto the screen.

• Show a first sign in under a second, even if it is only a status like “searching”.
• Do not force-scroll the page. If the reader scrolls up to reread, auto-scroll stops until they come back down.
• Render Markdown without flicker. A code block that is open but not yet closed should not turn everything after it into code: close unfinished blocks provisionally at render time.
• Plan for stopping. A Stop button that cancels the request and propagates the cancellation all the way to the model call, so you do not pay for tokens nobody will read.
• Handle the drop. Keep the text already received and offer to resume, rather than wiping it and showing an error.

Locally, streaming just works. In production it depends on every intermediary between the model and the screen, and it is the first thing I check on an environment that looks like prod.