AI

LLM Response Streaming: Server-Sent Events vs WebSockets

4 min read by DebuggedIt

Quick answer

Quick answer: for most LLM chat interfaces, Server-Sent Events are the simpler choice. They stream text one way over plain HTTP, reconnect automatically, and...

Quick answer: for most LLM chat interfaces, Server-Sent Events are the simpler choice. They stream text one way over plain HTTP, reconnect automatically, and pass through standard infrastructure. Choose WebSockets when you need true two-way messaging on one connection. In both cases, disable proxy buffering and cancel the upstream model call when the client disconnects.

Why Stream LLM Responses at All?

Models generate text gradually, and a full answer can take many seconds. Waiting for the complete response makes an application feel frozen, while streaming tokens as they arrive shows progress almost immediately. Perceived latency depends mostly on the time to the first visible text, so streaming improves the experience even when total generation time does not change.

Streaming also changes your architecture. Your server must keep a connection open for the whole generation, forward chunks as they arrive from the model provider, and handle failures in the middle of a response. Connection limits, timeouts, and memory use per request all matter more than in a short request-response API.

Decide what a partial answer means for your product. If you display text as it streams, users may see an answer that later fails or is cut off. Plan how to mark incomplete output, how to retry, and whether any post-processing, such as moderation or formatting, must happen before text is shown.

How Do SSE and WebSockets Compare?

Server-Sent Events use a normal HTTP response with a text event stream content type. The server sends events as lines of text, and the browser's built-in client parses them and reconnects after drops. They work well with existing HTTP tooling such as authentication headers, load balancers, and observability. They are one-way, so the client sends its request separately.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

data: {"delta": "Hello"}

data: {"delta": " world"}

data: [DONE]
  • SSE fits: chat responses, progress updates, and any case where the client asks once and the server streams back.
  • WebSockets fit: bidirectional flows such as voice, collaborative editing, or sending interruptions and tool results mid-stream on one channel.
  • Infrastructure: WebSockets need upgrade support in proxies, longer idle timeouts, and often sticky routing, which adds operational work.
  • Browser limits: over HTTP/1.1, browsers limit simultaneous connections per host, which can affect many open streams. HTTP/2 largely removes this.

The standard browser SSE client cannot send custom headers or a request body. Many applications use a fetch-based streaming reader or a library instead, which gives control over authentication and POST requests.

What Breaks Streaming in Production?

Buffering is the usual culprit. Reverse proxies, CDNs, and application servers may hold response data until a buffer fills, so users see nothing and then everything at once. Disable proxy buffering for streaming routes, and check compression settings, since some compression layers also buffer. The exact directives depend on your proxy, so consult its documentation.

Idle timeouts also cut long streams. If a model pauses for a while, a load balancer may close the connection. Send periodic comment lines as heartbeats, and set timeouts that match your longest expected generation.

Finally, handle cancellation. When a user closes the tab or presses stop, abort the upstream request to the model provider. Otherwise, you keep generating, and paying for, text no one will read. Propagate disconnect signals from the client connection to your upstream call and clean up promptly.

Frequently Asked Questions

Is SSE or WebSocket better for a chatbot?

SSE is usually better for a simple chat, because it is easier to deploy and debug. Use WebSockets only if you need two-way communication on the same connection.

Why does my stream arrive all at once?

A proxy or server is buffering the response. Disable buffering for that route and check compression and CDN settings.

How do I stop generation when the user clicks stop?

Abort the client request, detect the disconnect on the server, and cancel the upstream model call so you stop generating and paying for tokens.

Can I stream with POST requests?

Yes, using a fetch-based reader. The built-in SSE client supports only GET requests, so many apps use fetch to send a body.

Summary

Stream to reduce perceived latency, prefer SSE for one-way chat output, and use WebSockets for real two-way needs. Turn off buffering, send heartbeats, and cancel upstream calls on disconnect. Verify proxy and platform settings in their documentation.

You might also like