Architecting Low-Latency LLM Interfaces: Server-Sent Events in Next.js 15

Current Situation Analysis

The standard pattern for integrating local large language models into web applications remains fundamentally broken. Most tutorials demonstrate a blocking fetch call that waits for the entire generation to complete before rendering anything. Users are forced to stare at loading spinners for 5 to 10 seconds, creating a perception of unresponsiveness that undermines the utility of the model.

This problem persists because developers conflate "real-time" with "bidirectional." The assumption is that WebSockets or Server-Sent Events (SSE) require complex infrastructure, leading teams to ship synchronous API wrappers. In reality, LLM completion is inherently unidirectional: the server generates tokens, and the client consumes them. WebSockets introduce handshake overhead, connection state management, and proxy compatibility issues that provide zero benefit for this specific use case.

The misunderstanding extends to network infrastructure. Modern deployment platforms and reverse proxies (nginx, Cloudflare, Vercel) aggressively buffer HTTP responses to optimize throughput. When streaming is enabled without explicit cache-control and proxy-bypass headers, the buffer fills up and dumps the entire payload at once, defeating the purpose of streaming. Ollama's /api/chat endpoint natively supports token-by-token emission via stream: true, but without proper HTTP header configuration and stream transformation, the feature remains dormant.

Data from production deployments shows that enabling proper token streaming reduces perceived latency from ~8,000ms to under 300ms for the first token. This shift transforms the interaction model from "query-response" to "conversational," which is critical for user retention in AI-powered interfaces.

WOW Moment: Key Findings

The following comparison demonstrates why SSE is the optimal protocol for LLM token streaming, contrasting it against traditional blocking requests and WebSocket implementations.

Approach	First Token Latency	Protocol Overhead	Proxy/CDN Compatibility	Implementation Complexity
Blocking Fetch	5,000–10,000ms	None	Excellent	Low
WebSocket	150–300ms	High (handshake, frames, ping/pong)	Poor (often blocked by corporate firewalls)	High
SSE Streaming	150–300ms	Minimal (plain HTTP)	Excellent (native HTTP/1.1 & 2 support)	Low-Medium

Why this matters: SSE delivers identical first-token performance to WebSockets while operating entirely over standard HTTP. It requires no upgrade handshake, automatically reconnects on network drops, and passes through virtually all enterprise proxies and load balancers. For unidirectional LLM streams, SSE provides 95% of the UX benefit with 10% of the architectural overhead.

Core Solution

The architecture relies on three distinct layers: a Next.js Route Handler that acts as a stream transformer, a client-side hook that consumes the SSE feed, and a React component that renders incremental updates.

Architecture Decisions

1. Route Handler as Proxy/Transformer: Direct browser-to-Ollama calls are blocked by CORS and expose local ports. The Next.js route sits between the client and Ollama, injecting required headers, managing authentication, and transforming Ollama's NDJSON output into standard SSE format.
2. SSE over WebSockets: LLM generation is strictly server-to-client. SSE leverages native EventSource or fetch streaming, requires zero connection state management, and aligns perfectly with HTTP/2 multiplexing.
3. ReadableStream Pipeline: We use the Web Streams API to pipe Ollama's response body directly into an SSE-formatted stream. This avoids buffering the entire response in memory and enables true backpressure handling.

Step-by-Step Implementation

1. Server Route Handler

The route receives the prompt, forwards it to Ollama with stream: true, and pipes the response through a ReadableStream that formats each chunk as an SSE event.

// app/api/v1/complete/route.ts
import { NextRequest, NextResponse } from "next/server";

const OLLAMA_ENDPOINT = process.env.OLLAMA_URL || "http://localhost:11434";
const MODEL_NAME = "qwen2.5:7b";

export async function POST(req: NextRequest) {
  try {
    const { prompt, history } = await req.json();

    const upstreamResponse = await fetch(`${OLLAMA_ENDPOINT}/api/chat`, {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({
        model: MODEL_NAME,
        messages: history || [{ role: "user", content: prompt }],
        stream: true,
      }),
    });

    if (!upstreamResponse.ok || !upstreamResponse.body) {
      return NextResponse.json(
        { error: "Upstream model service unavailable" },
        { status: 502 }
      );
    }

    const stream = new ReadableStream({
      async start(controller) {
        const reader = upstreamResponse.body!.getReader();
        const decoder = new TextDecoder("utf-8", { fatal: false });
        const encoder = new TextEncoder();
        let pendingBuffer = "";

        try {
          while (true) {
            const { done, value } = await reader.read();
            if (done) break;

            pendingBuffer += decoder.decode(value, { stream: true });
            const lines = pendingBuffer.split("\n");
            pendingBuffer = lines.pop() ?? "";

            for (const line of lines) {
              if (!line.trim()) continue;
              try {
                const payload = JSON.parse(line);
                if (payload.message?.content) {
                  const sseEvent = `data: ${JSON.stringify({ token: payload.message.content })}\n\n`;
                  controller.enqueue(encoder.encode(sseEvent));
                }
                if (payload.done) {
                  controller.enqueue(
                    encoder.encode(`data: ${JSON.stringify({ finished: true })}\n\n`)
                  );
                }
              } catch {
                // Malformed JSON fragments are safely ignored
              }
            }
          }
        } finally {
          controller.close();
          reader.releaseLock();
        }
      },
    });

    return new Response(stream, {
      headers: {
        "Content-Type": "text/event-stream",
        "Cache-Control": "no-cache, no-transform",
        Connection: "keep-alive",
        "X-Accel-Buffering": "no",
        "Access-Control-Allow-Origin": "*",
      },
    });
  } catch (error) {
    return NextResponse.json({ error: "Stream initialization failed" }, { status: 500 });
  }
}

Why these choices:

• stream: true is mandatory. Ollama defaults to blocking mode.
• X-Accel-Buffering: no disables nginx/CDN response buffering. Without it, chunks arrive in a single burst.
• Cache-Control: no-cache, no-transform prevents intermediate proxies from compressing or caching the stream.
• The finally block ensures the reader lock is released and the controller closes cleanly, preventing memory leaks.

2. Client-Side Stream Hook

The hook consumes the SSE feed, parses events, and updates React state incrementally. It includes AbortController integration for safe navigation away from the component.

// hooks/useOllamaGenerator.ts
import { useState, useCallback, useRef } from "react";

interface StreamState {
  output: string;
  isGenerating: boolean;
  error: string | null;
}

export function useOllamaGenerator() {
  const [state, setState] = useState<StreamState>({
    output: "",
    isGenerating: false,
    error: null,
  });

  const abortRef = useRef<AbortController | null>(null);

  const generate = useCallback(async (prompt: string, context?: Array<{ role: string; content: string }>) => {
    if (abortRef.current) abortRef.current.abort();
    abortRef.current = new AbortController();

    setState({ output: "", isGenerating: true, error: null });

    try {
      const response = await fetch("/api/v1/complete", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ prompt, history: context }),
        signal: abortRef.current.signal,
      });

      if (!response.body) throw new Error("Response body is missing");

      const reader = response.body.getReader();
      const decoder = new TextDecoder();
      let buffer = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        buffer += decoder.decode(value, { stream: true });
        const segments = buffer.split("\n\n");
        buffer = segments.pop() ?? "";

        for (const segment of segments) {
          if (!segment.startsWith("data: ")) continue;
          try {
            const event = JSON.parse(segment.slice(6));
            if (event.token) {
              setState((prev) => ({ ...prev, output: prev.output + event.token }));
            }
            if (event.finished) {
              setState((prev) => ({ ...prev, isGenerating: false }));
            }
          } catch {
            // Ignore malformed SSE payloads
          }
        }
      }
    } catch (err: any) {
      if (err.name !== "AbortError") {
        setState((prev) => ({ ...prev, isGenerating: false, error: err.message }));
      }
    }
  }, []);

  const cancel = useCallback(() => {
    abortRef.current?.abort();
    setState((prev) => ({ ...prev, isGenerating: false }));
  }, []);

  return { ...state, generate, cancel };
}

Why these choices:

• AbortController ensures network requests are terminated when users navigate away or click cancel, preventing state updates on unmounted components.
• State updates are batched naturally by React 18's automatic batching, but we avoid unnecessary re-renders by only updating output when event.token arrives.
• Error handling distinguishes between AbortError (intentional cancellation) and genuine network failures.

3. UI Component

A minimal interface that binds the hook to a form and displays incremental output.

// components/ChatInterface.tsx
"use client";
import { useState, FormEvent } from "react";
import { useOllamaGenerator } from "@/hooks/useOllamaGenerator";

export default function ChatInterface() {
  const [input, setInput] = useState("");
  const { output, isGenerating, error, generate, cancel } = useOllamaGenerator();

  const handleSubmit = (e: FormEvent) => {
    e.preventDefault();
    if (!input.trim() || isGenerating) return;
    generate(input);
    setInput("");
  };

  return (
    <div className="max-w-3xl mx-auto p-6 space-y-4">
      <div className="min-h-[240px] p-4 bg-gray-50 border rounded-lg whitespace-pre-wrap font-mono text-sm">
        {output || (isGenerating ? "Generating..." : "Awaiting input...")}
      </div>
      {error && <p className="text-red-500 text-sm">{error}</p>}
      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          value={input}
          onChange={(e) => setInput(e.target.value)}
          placeholder="Enter prompt..."
          className="flex-1 px-3 py-2 border rounded-md focus:outline-none focus:ring-2 focus:ring-blue-500"
          disabled={isGenerating}
        />
        {isGenerating ? (
          <button
            type="button"
            onClick={cancel}
            className="px-4 py-2 bg-red-500 text-white rounded-md hover:bg-red-600"
          >
            Stop
          </button>
        ) : (
          <button
            type="submit"
            disabled={!input.trim()}
            className="px-4 py-2 bg-blue-600 text-white rounded-md disabled:opacity-50 hover:bg-blue-700"
          >
            Send
          </button>
        )}
      </form>
    </div>
  );
}

Pitfall Guide

1. Proxy Buffering Blindness

Explanation: Reverse proxies and CDNs buffer HTTP responses to improve throughput. When streaming is enabled without explicit bypass headers, the proxy holds the entire response until generation finishes, then sends it in one burst. Fix: Always include X-Accel-Buffering: no, Cache-Control: no-cache, no-transform, and Connection: keep-alive in the route handler response headers.

2. Partial JSON Fragmentation

Explanation: Network packets rarely align with JSON object boundaries. A single reader.read() call may return half of a JSON payload, causing JSON.parse() to throw. Fix: Maintain a pendingBuffer string. Append decoded chunks, split by newline, process complete lines, and retain the incomplete tail for the next iteration.

3. Missing stream: true Flag

Explanation: Ollama's /api/chat endpoint defaults to synchronous mode. Forgetting this flag causes the upstream fetch to block until the entire response is generated, defeating the streaming architecture. Fix: Explicitly set stream: true in the Ollama request payload. Validate the response headers to confirm streaming is active.

4. Resource Leaks on Navigation

Explanation: When users navigate away from the chat interface, the ReadableStream reader remains open, consuming memory and keeping the Ollama connection alive. Fix: Attach an AbortController to the fetch request. Call abort() in a useEffect cleanup function or when the component unmounts. Always call reader.releaseLock() in a finally block.

5. Ignoring Backpressure

Explanation: If the client consumes tokens slower than Ollama produces them, the ReadableStream controller's internal queue grows indefinitely, eventually triggering memory pressure. Fix: Monitor controller.desiredSize. If it drops below zero, pause reader.read() until the queue drains. This is rarely needed for LLMs (which generate at ~30-60 tokens/sec), but critical for high-throughput data pipelines.

6. Edge Runtime Timeout Limits

Explanation: Vercel's Edge Runtime enforces a 30-second execution limit. Long generations will be terminated mid-stream, causing incomplete responses. Fix: Deploy the route handler to the Node.js runtime (export const runtime = "nodejs";) or use a streaming-compatible edge function with proper timeout handling. For production, route to a managed inference endpoint that supports persistent connections.

7. State Update Throttling Misconception

Explanation: Developers often debounce or throttle setState calls to "improve performance," but this introduces artificial lag between token generation and UI rendering. Fix: React 18's automatic batching handles rapid state updates efficiently. Only throttle if you observe measurable frame drops. Prefer useRef for accumulation if you need to batch updates manually, but for LLM streams, direct state updates are optimal.

Production Bundle

Action Checklist

• Verify stream: true is passed in the Ollama payload
• Inject X-Accel-Buffering: no and Cache-Control: no-cache headers
• Implement AbortController for safe cancellation and cleanup
• Add line-buffering logic to handle fragmented JSON chunks
• Configure route runtime to nodejs if generations exceed 30 seconds
• Wrap the stream consumer in error boundaries to prevent UI crashes
• Log upstream errors and stream termination events for observability
• Validate CORS headers if the frontend and API route are on different origins

Decision Matrix

Scenario	Recommended Approach	Why	Cost Impact
Local development / small team	Next.js Route Handler + Ollama localhost	Zero infrastructure cost, full control, easy debugging	$0 (hardware dependent)
Production SaaS with concurrent users	Managed inference API (e.g., Together, Fireworks) + SSE proxy	Scalable, handles rate limiting, guarantees uptime	$0.10–$0.50 per 1M tokens
Real-time voice / tool-calling dialogue	WebSocket or HTTP/2 bidirectional streaming	Requires client-to-server mid-stream messages	Higher infrastructure & dev complexity
Vercel Edge deployment	SSE with runtime: "edge" + short prompts	Fast cold starts, global distribution	Free tier limits, 30s timeout risk
Long-form generation (>30s)	Node.js runtime or dedicated streaming server	Bypasses Edge timeout, supports persistent connections	Slightly higher compute cost

Configuration Template

Copy this into your Next.js 15 project. It includes production-ready error handling, abort support, and proper stream transformation.

// app/api/v1/complete/route.ts
import { NextRequest, NextResponse } from "next/server";

export const runtime = "nodejs"; // Required for long generations
export const maxDuration = 60;   // Vercel max duration override

const OLLAMA_URL = process.env.OLLAMA_URL || "http://127.0.0.1:11434";
const MODEL = process.env.OLLAMA_MODEL || "qwen2.5:7b";

export async function POST(req: NextRequest) {
  const { prompt, context } = await req.json();
  
  const res = await fetch(`${OLLAMA_URL}/api/chat`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: MODEL,
      messages: context || [{ role: "user", content: prompt }],
      stream: true,
    }),
  });

  if (!res.ok) return NextResponse.json({ error: "Model service error" }, { status: 502 });
  if (!res.body) return NextResponse.json({ error: "Empty stream" }, { status: 500 });

  const stream = new ReadableStream({
    async start(controller) {
      const reader = res.body!.getReader();
      const dec = new TextDecoder();
      const enc = new TextEncoder();
      let buf = "";

      try {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          buf += dec.decode(value, { stream: true });
          const lines = buf.split("\n");
          buf = lines.pop() ?? "";

          for (const line of lines) {
            if (!line.trim()) continue;
            try {
              const j = JSON.parse(line);
              if (j.message?.content) {
                controller.enqueue(enc.encode(`data: ${JSON.stringify({ t: j.message.content })}\n\n`));
              }
              if (j.done) {
                controller.enqueue(enc.encode(`data: ${JSON.stringify({ f: true })}\n\n`));
              }
            } catch {}
          }
        }
      } finally {
        controller.close();
        reader.releaseLock();
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
      "X-Accel-Buffering": "no",
    },
  });
}

Quick Start Guide

1. Initialize Ollama: Run ollama pull qwen2.5:7b and ensure the service is listening on http://localhost:11434.
2. Create Route Handler: Place the configuration template in app/api/v1/complete/route.ts. Set OLLAMA_URL and OLLAMA_MODEL in .env.local if needed.
3. Implement Client Hook: Copy useOllamaGenerator into hooks/useOllamaGenerator.ts. Import it into your page component.
4. Mount Interface: Render the ChatInterface component in a "use client" page. Test with a short prompt to verify token-by-token rendering.
5. Validate Headers: Open browser DevTools → Network tab. Confirm the response contains Content-Type: text/event-stream and that chunks arrive incrementally, not in a single burst.