Ever wondered why a chatbot feels sluggish when it has to travel across continents before you see the first word?
Why stream LLM responses from the edge?
Large language models (LLMs) such as GPT‑4 generate output token by token. Traditional HTTP responses wait until the whole answer is ready, adding latency that users notice as a blank screen. By streaming each token as soon as the model produces it, you can render partial answers in real time. Deploying the streaming logic on an edge platform like Cloudflare Workers reduces round‑trip time to under 30 ms for European users and under 80 ms for North America, according to Cloudflare’s 2024 performance report. The combination of edge computing, JavaScript, and Server‑Sent Events (SSE) creates a low‑cost, globally distributed chatbot that feels instantaneous.
Setting up a Cloudflare Worker for SSE
Cloudflare Workers run JavaScript V8 isolates at the edge, so you can call the OpenAI API, transform the raw streaming payload, and re‑emit it as SSE without any server. First, add your OpenAI key to the Worker environment variables (e.g., OPENAI_API_KEY). Then implement a handler that accepts a prompt query parameter, forwards the request to the OpenAI /chat/completions endpoint with stream:true, and pipes the incoming chunks into an SSE‑compatible format.
addEventListener('fetch', event => { event.respondWith(handleRequest(event.request)); }); async function handleRequest(request) { const url = new URL(request.url); const prompt = url.searchParams.get('prompt') || 'Hello'; const apiResponse = await fetch('https://api.openai.com/v1/chat/completions', { method: 'POST', headers: { 'Content-Type': 'application/json', 'Authorization': `Bearer ${OPENAI_API_KEY}` }, body: JSON.stringify({ model: 'gpt-4o-mini', messages: [{ role: 'user', content: prompt }], stream: true }) }); const { readable, writable } = new TransformStream(); const encoder = new TextEncoder(); apiResponse.body.pipeTo(new WritableStream({ async write(chunk) { const text = new TextDecoder().decode(chunk); const lines = text.split('\\n'); for (const line of lines) { if (line.startsWith('data:')) { const json = JSON.parse(line.slice(5)); const delta = json.choices[0].delta?.content; if (delta) { const sse = `data: ${delta}\\n\\n`; writable.getWriter().write(encoder.encode(sse)); } } } } })); return new Response(readable, { headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive' } }); }The TransformStream bridges the OpenAI byte stream to a readable stream that the Worker returns as text/event-stream. Each token arrives as a separate data: line, which browsers interpret as a message event.
JavaScript fetch and event handling on the client
On the front end, a simple EventSource object consumes the SSE endpoint. Because SSE works over GET, the prompt is passed as a query string. The client appends each incoming token to a DOM element, giving the impression of a live‑typing assistant.
async function ask(prompt) { const evtSource = new EventSource(`/chat?prompt=${encodeURIComponent(prompt)}`); let answer = ''; evtSource.onmessage = e => { answer += e.data; document.getElementById('output').textContent = answer; }; evtSource.onerror = () => { evtSource.close(); }; }Integrate the function with a textarea and a button, and you have a fully functional edge‑hosted chatbot with sub‑second perceived latency.
Performance tips and cost considerations
1. **Cache static assets** – Cloudflare automatically caches HTML, CSS, and JS at the edge, keeping the only dynamic request the SSE stream. 2. **Limit token length** – Set max_tokens to 150 in the OpenAI payload to avoid runaway costs; at $0.00015 per token, a 150‑token answer costs roughly $0.0225. 3. **Use Workers Bundling** – Deploy with wrangler publish --minify to shrink the JavaScript bundle, cutting cold‑start time to under 5 ms. 4. **Monitor latency** – Cloudflare’s Dashboard shows per‑region response times; aim for < 100 ms for best UX. By keeping the entire pipeline on the edge, you eliminate the need for a separate backend server, reducing operational overhead to a few dollars per month for moderate traffic.
Conclusion
Streaming LLM responses from the edge combines three powerful trends: JavaScript‑native serverless compute, Server‑Sent Events for real‑time delivery, and the global reach of Cloudflare Workers. The result is a chatbot that feels local, responsive, and cost‑effective. Implement the code snippets above, tune token limits, and watch your users engage instantly with AI‑driven conversation.
Sources
Cloudflare Workers Documentation, OpenAI API Reference, MDN Web Docs – Server‑Sent Events
Author: Mahmut Sarıkaya — sarikayadev.com