Streaming Responses Guide

This guide covers best practices for handling streaming responses across different platforms and languages.

Why stream?

Better UX: Users see tokens appear in real time instead of waiting 2-5 seconds for a full response
Lower perceived latency: Users see the first token as soon as the model starts generating (latency varies by model and region)
Memory efficient: Process tokens as they arrive instead of buffering the full response

Python (OpenAI SDK)

stream = client.chat.completions.create(
    model="gpt-oss-20b",
    messages=[{"role": "user", "content": "Explain transformers"}],
    stream=True
)
 
full_response = ""
for chunk in stream:
    if not chunk.choices:  # usage-only chunk at the end of the stream
        continue
    content = chunk.choices[0].delta.content or ""
    full_response += content
    print(content, end="", flush=True)

Python (async)

import asyncio
 
async def stream_response():
    stream = await async_client.chat.completions.create(
        model="gpt-oss-20b",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True
    )
 
    async for chunk in stream:
        if not chunk.choices:  # usage-only chunk at the end of the stream
            continue
        content = chunk.choices[0].delta.content or ""
        print(content, end="", flush=True)
 
asyncio.run(stream_response())

Node.js

const stream = await client.chat.completions.create({
  model: 'gpt-oss-20b',
  messages: [{ role: 'user', content: 'Hello' }],
  stream: true,
});
 
for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || '';
  process.stdout.write(content);
}

This component expects your own /api/chat route to return plain text — for example, a route that calls SovereignEG with the OpenAI SDK and writes out only each chunk's delta.content. If the route passes SovereignEG's server-sent events through unchanged, parse them instead of appending raw text: keep a buffer across reads, split it on blank lines, parse the JSON after each data: prefix (an event with an error object means the stream failed), and stop at data: [DONE].

'use client'
import { useState } from 'react'
 
export function Chat() {
  const [output, setOutput] = useState('')
  const [loading, setLoading] = useState(false)
 
  async function handleSubmit(prompt: string) {
    setLoading(true)
    setOutput('')
 
    const response = await fetch('/api/chat', {
      method: 'POST',
      body: JSON.stringify({ prompt }),
    })
 
    const reader = response.body!.getReader()
    const decoder = new TextDecoder()
 
    while (true) {
      const { done, value } = await reader.read()
      if (done) break
 
      const text = decoder.decode(value, { stream: true })
      setOutput(prev => prev + text)
    }
 
    setLoading(false)
  }
 
  return <div>{output}{loading && <span className="animate-pulse">▊</span>}</div>
}

Error handling

A streaming request can fail in two places. Before the stream starts it fails with a normal HTTP error status, so the create call raises. A 429 inflight_busy here means your organization's other requests still running hold the credit this one needs (the API already waited up to about 25 seconds for them): wait for Retry-After (10 s) and send it again; the OpenAI SDKs retry a 429 by themselves, twice by default. Once the stream has started, the error arrives inside the stream, or the connection drops, and it is raised while you iterate. Handle both:

from openai import APIConnectionError, APIStatusError
 
try:
    stream = client.chat.completions.create(
        model="gpt-oss-20b",
        messages=[{"role": "user", "content": "Hello"}],
        stream=True,
    )
except APIStatusError as e:
    # Failed before the stream started: e.status_code is the HTTP status
    print(f"Request failed: {e.status_code} {e.message}")
except APIConnectionError as e:
    print(f"Could not reach the API: {e}")
else:
    try:
        for chunk in stream:
            if not chunk.choices:  # usage-only chunk at the end of the stream
                continue
            content = chunk.choices[0].delta.content or ""
            process(content)
    except Exception as e:
        # An error event inside the stream, or a dropped connection
        print(f"Stream interrupted: {e}")
        # Optionally retry or fall back to non-streaming

Best practices

Always flush output — use flush=True in Python or process.stdout.write in Node
Show a cursor — display a blinking cursor while streaming for better UX
Handle [DONE] — the stream ends with data: [DONE]; your parser must handle this
Set generous timeouts — the API sends no bytes (not even response headers) until the model's first streamed chunk, which can take longer than 30 seconds for long prompts or reasoning models. Allow up to 5 minutes, the API's own limit, before giving up. A request you abandon is still billed for the work already done, and the OpenAI SDKs retry a timed-out request twice by default; each retry is a new request that counts toward your rate limit and is billed like any other
Buffer by word — for display, buffer until a space character for smoother text rendering
Handle a balance stop — when your organization's spendable credit runs out part-way, the stream is not dropped: its last chunk has finish_reason: "length", a usage block with the tokens you are billed for and an extra top-level stop_reason: "balance_exhausted", then data: [DONE] (see Streaming). Keep the text, and send the next request after a top-up or once your other requests finish. A few models measured ignoring max_tokens end the same way with stop_reason: "cap_exceeded" instead: that is not about credit, and output past your max_tokens is not billed. A stream that reaches your API key's or project's monthly spend cap ends the same way with stop_reason: "spend_cap_reached": raise the cap, or wait for the month to reset, rather than topping up